Oct 30 17:12:02.730031 ip-10-0-135-170 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 17:12:03.145958 ip-10-0-135-170 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 17:12:03.145958 ip-10-0-135-170 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 17:12:03.145958 ip-10-0-135-170 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 17:12:03.145958 ip-10-0-135-170 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 17:12:03.149635 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149529    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.149635 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149634    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.149635 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149652    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149657    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149660    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149663    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149666    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149669    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149672    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149674    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149677    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149680    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149683    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149685    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149688    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149691    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149694    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149697    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149699    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149704    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149708    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.149747 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149712    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149714    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149717    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149720    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149723    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149732    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149735    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149739    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149741    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149744    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149746    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149749    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149752    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149755    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149758    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149761    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149764    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149767    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149769    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.150190 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149772    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149775    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149777    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149780    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149782    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149785    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149787    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149790    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149792    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149795    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149799    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149802    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149805    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149808    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149811    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149815    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149817    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149820    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149823    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149826    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.150628 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149828    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149831    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149834    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149836    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149839    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149842    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149844    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149847    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149849    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149852    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149854    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149857    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149859    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149862    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149865    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149868    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149870    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149872    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149876    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.151153 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149879    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149881    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149884    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149886    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149889    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149893    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149895    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149898    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149901    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149903    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149906    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149908    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149911    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149913    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149916    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149919    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149921    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149924    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149928    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149931    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.151613 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149933    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149936    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149938    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149941    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149943    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149947    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.149954    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150840    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150850    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150853    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150857    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150863    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150867    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150870    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150873    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150876    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150878    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150881    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.152108 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150884    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150887    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150890    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150893    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150896    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150898    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150901    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150904    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150907    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150909    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150912    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150914    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150917    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150920    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150922    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150925    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150928    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150930    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150933    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150935    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.152544 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150938    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150940    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150943    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150946    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150949    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150952    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150954    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150957    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150960    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150963    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150965    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150969    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150972    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150974    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150978    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150981    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150984    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150987    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150989    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.153029 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150992    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150994    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.150999    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151003    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151006    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151009    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151011    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151015    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151017    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151020    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151022    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151025    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151027    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151030    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151032    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151035    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151038    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151041    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151044    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151050    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.153486 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151053    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151057    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151060    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151063    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151065    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151068    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151071    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151073    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151076    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151078    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151081    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151083    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151086    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151088    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151091    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151093    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151096    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151098    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151101    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151104    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.154013 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151107    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151110    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151112    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151115    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151117    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151120    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151122    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151125    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151128    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151130    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151133    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151137    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151139    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151142    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151150    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151154    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151730    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151737    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151740    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151743    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.154491 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151746    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151748    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151751    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151754    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151757    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151759    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151762    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151765    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151767    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151771    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151773    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151778    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151781    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151783    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151786    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151789    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151791    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151794    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151797    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151799    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.154979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151802    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151804    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151807    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151810    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151813    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151816    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151819    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151822    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151825    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151827    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151830    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151832    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151835    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151837    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151840    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151843    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151845    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151848    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151850    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151853    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.155573 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151855    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151858    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151861    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151865    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151868    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151870    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151873    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151875    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151878    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151880    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151885    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151888    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151891    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151894    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151896    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151899    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151902    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151905    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151907    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151910    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.156069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151913    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151915    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151918    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151920    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151924    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151929    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151931    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151934    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151937    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151939    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151942    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151945    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151948    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151951    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151953    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151958    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151960    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151963    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151966    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.156533 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151969    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151972    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151974    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151977    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151979    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151982    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151984    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151987    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151989    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151992    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151994    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151997    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.151999    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152003    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152005    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152008    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152011    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152013    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152016    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.156992 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152019    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152022    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152025    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.152028    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152900    2570 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152918    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152925    2570 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152931    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152937    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152941    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152946    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152951    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152955    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152958    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152962    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152965    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152968    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152972    2570 flags.go:64] FLAG: --cgroup-root=""
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152974    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152978    2570 flags.go:64] FLAG: --client-ca-file=""
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152980    2570 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152984    2570 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152989    2570 flags.go:64] FLAG: --cluster-domain=""
Oct 30 17:12:03.157444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152992    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152996    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.152999    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153003    2570 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153008    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153011    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153015    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153018    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153021    2570 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153024    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153028    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153031    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153034    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153039    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153043    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153048    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153051    2570 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153054    2570 flags.go:64] FLAG: --enable-server="true"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153057    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153062    2570 flags.go:64] FLAG: --event-burst="100"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153066    2570 flags.go:64] FLAG: --event-qps="50"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153069    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153072    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153076    2570 flags.go:64] FLAG: --eviction-hard=""
Oct 30 17:12:03.157996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153080    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153083    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153087    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153090    2570 flags.go:64] FLAG: --eviction-soft=""
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153093    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153096    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153099    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153102    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153105    2570 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153108    2570 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153111    2570 flags.go:64] FLAG: --feature-gates=""
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153116    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153119    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153128    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153132    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153136    2570 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153139    2570 flags.go:64] FLAG: --help="false"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153142    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153145    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153148    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153152    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153155    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153159    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153164    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 17:12:03.158609 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153167    2570 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153170    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153173    2570 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153176    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153179    2570 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153182    2570 flags.go:64] FLAG: --kube-reserved=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153185    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153188    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153192    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153195    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153198    2570 flags.go:64] FLAG: --lock-file=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153201    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153204    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153207    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153213    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153216    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153219    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153222    2570 flags.go:64] FLAG: --logging-format="text"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153225    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153229    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153233    2570 flags.go:64] FLAG: --manifest-url=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153236    2570 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153241    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153245    2570 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153249    2570 flags.go:64] FLAG: --max-pods="110"
Oct 30 17:12:03.159189 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153253    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153256    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153259    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153262    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153265    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153268    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153272    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153280    2570 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153284    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153288    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153291    2570 flags.go:64] FLAG: --pod-cidr=""
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153294    2570 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153297    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153300    2570 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153303    2570 flags.go:64] FLAG: --port="10250"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153306    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153309    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7b45970d80d2ce6"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153312    2570 flags.go:64] FLAG: --qos-reserved=""
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153315    2570 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153319    2570 flags.go:64] FLAG: --register-node="true"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153321    2570 flags.go:64] FLAG: --register-with-taints=""
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153326    2570 flags.go:64] FLAG: --registry-burst="10"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153329    2570 flags.go:64] FLAG: --registry-qps="5"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153332    2570 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153335    2570 flags.go:64] FLAG: --reserved-memory=""
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153338    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 17:12:03.159874 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153342    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153345    2570 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153348    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153351    2570 flags.go:64] FLAG: --runonce="false"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153354    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153357    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153360    2570 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153363    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153366    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153370    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153373    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153376    2570 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153382    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153386    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153389    2570 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153392    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153395    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153398    2570 flags.go:64] FLAG: --system-cgroups=""
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153401    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153407    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153410    2570 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153414    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153421    2570 flags.go:64] FLAG: --tls-min-version=""
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153424    2570 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153427    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 17:12:03.160852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153431    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153434    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153438    2570 flags.go:64] FLAG: --v="2"
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153442    2570 flags.go:64] FLAG: --version="false"
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153447    2570 flags.go:64] FLAG: --vmodule=""
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153451    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153455    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153598    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153602    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153605    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153608    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153611    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153614    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153617    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153620    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153623    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153625    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153628    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153630    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153636    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153651    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.161926 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153654    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153657    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153659    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153662    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153665    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153668    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153673    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153677    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153681    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153684    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153687    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153691    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153693    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153696    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153700    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153702    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153705    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153707    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153710    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153713    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.162824 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153715    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153719    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153721    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153724    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153727    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153730    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153733    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153735    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153738    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153741    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153747    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153749    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153752    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153755    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153757    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153760    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153762    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153765    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153768    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153770    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.163724 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153773    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153776    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153778    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153781    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153783    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153786    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153788    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153791    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153793    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153796    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153799    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153802    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153804    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153807    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153809    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153812    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153815    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153818    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153820    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.164670 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153823    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153825    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153828    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153833    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153836    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153839    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153844    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153847    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153850    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153853    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153855    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153858    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153861    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153863    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153866    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153869    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153871    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153874    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153876    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.165254 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153879    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153882    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153884    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153887    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153889    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153892    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153894    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153896    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153899    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153902    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153904    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153907    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153910    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.153913    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.153969    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154010    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.165899 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154016    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154021    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154025    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154029    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154031    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154034    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154037    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154040    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154043    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154046    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154049    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154051    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154054    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154057    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154059    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154062    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154064    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154087    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154092    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.166549 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154097    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154100    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154103    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154105    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154108    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154111    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154113    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154116    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154118    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154121    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154124    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154127    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154130    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154134    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154138    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154141    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154143    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154146    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154149    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154151    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.167069 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154154    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154157    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154159    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154162    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154165    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154167    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154170    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154172    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154175    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154177    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154180    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154182    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154185    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154188    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154191    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154193    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154196    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154198    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154201    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154203    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.167561 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154206    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154208    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154211    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154214    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154217    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154221    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154225    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154228    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154231    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154233    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154236    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154238    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154241    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154244    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154246    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154249    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154251    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154255    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154258    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.168103 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154261    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154264    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154266    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154269    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154271    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154274    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154276    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154279    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154281    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154284    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154286    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154289    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154291    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154294    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154296    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154299    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154301    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154304    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154309    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.168551 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154313    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154315    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154318    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154321    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154323    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154326    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154328    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.154331    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.162665    2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.162694    2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.162720    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.162729    2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162822    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162831    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162838    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162843    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162847    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162852    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162855    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.169296 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162860    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162865    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162870    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162876    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162882    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162887    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162893    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162899    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162905    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162909    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162914    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162918    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162923    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162927    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162931    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162935    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162939    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162944    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162948    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.169913 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162953    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162957    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162961    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162973    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162978    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162982    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162987    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162991    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.162995    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163000    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163004    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163008    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163013    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163017    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163021    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163025    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163030    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163035    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163039    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163043    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.170470 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163049    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163053    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163057    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163064    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163071    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163077    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163083    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163087    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163092    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163096    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163101    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163105    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163110    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163114    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163119    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163123    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163130    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163135    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163140    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.171091 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163144    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163148    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163153    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163157    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163162    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163166    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163170    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163174    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163178    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163184    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163188    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163192    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163196    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163201    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163206    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163211    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163215    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163219    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163223    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.171898 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163227    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163231    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163235    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163240    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163245    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163250    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163255    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163259    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163264    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163268    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163272    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163277    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163281    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163286    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163290    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163294    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163299    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163304    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163308    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163312    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.172498 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163316    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163321    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.163372    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163610    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163619    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163624    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163630    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163635    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163659    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163664    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163669    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163673    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163677    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163682    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163686    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163690    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.172994 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163695    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163700    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163704    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163708    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163712    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163717    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163721    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163725    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163730    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163734    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163738    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163742    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163746    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163750    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163757    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163764    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163769    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163773    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163777    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163782    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.173371 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163786    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163790    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163794    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163799    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163803    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163808    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163812    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163817    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163821    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163826    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163830    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163835    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163839    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163843    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163847    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163852    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163856    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163860    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163865    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163869    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.173854 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163873    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163880    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163887    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163892    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163897    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163901    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163906    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163910    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163915    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163920    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163924    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163928    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163932    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163937    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163941    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163945    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163950    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163954    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163958    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.174417 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163962    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163967    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163971    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163975    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163984    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163989    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163993    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.163998    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164002    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164006    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164010    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164014    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164019    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164023    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164027    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164033    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164037    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164041    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164046    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.174928 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164050    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164054    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164058    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164062    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164067    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164071    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164076    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164080    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164084    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164088    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164092    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164096    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164101    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164105    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164109    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.164156    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 17:12:03.175406 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164213    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164221    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164226    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164230    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164234    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164241    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164247    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164253    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164257    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164263    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164267    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164272    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164276    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164282    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164287    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164291    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164297    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164301    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164306    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164310    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:12:03.175820 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164314    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164319    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164323    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164327    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164331    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164336    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164339    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164344    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164348    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164353    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164357    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164361    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164365    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164371    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164375    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164379    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164384    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164388    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164392    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:12:03.176290 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164397    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164401    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164405    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164409    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164414    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164418    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164422    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164427    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164432    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164438    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164442    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164447    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164451    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164455    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164459    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164463    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164467    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164471    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164475    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164480    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:12:03.176800 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164484    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164488    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164493    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164496    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164500    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164504    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164510    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164514    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164518    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164523    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164528    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164534    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164539    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164543    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164548    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164552    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164556    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164560    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164564    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164568    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:12:03.177271 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164573    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164579    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164584    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164589    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164593    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164600    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164606    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164610    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164615    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164619    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164624    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164628    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164632    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164637    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164659    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164663    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164667    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164672    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164677    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:12:03.177761 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164682    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164686    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164690    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164696    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164701    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164705    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164709    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:03.164714    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.164813    2570 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.169859    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.171000    2570 server.go:1014] "Starting client certificate rotation"
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.171144    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.171195    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.175253    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 17:12:03.178193 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.176511    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 17:12:03.189258 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.189240    2570 log.go:25] "Validated CRI v1 runtime API"
Oct 30 17:12:03.193260 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.193242    2570 log.go:25] "Validated CRI v1 image API"
Oct 30 17:12:03.194195 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.194168    2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 17:12:03.197785 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.197761    2570 fs.go:135] Filesystem UUIDs: map[01203fc1-c7d6-4797-b511-f669187fe360:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 93c6ddc4-ffdf-416a-94af-e9f37a28134c:/dev/nvme0n1p4]
Oct 30 17:12:03.197882 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.197782    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 17:12:03.203856 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.203721    2570 manager.go:217] Machine: {Timestamp:2026-10-30 17:12:03.201519929 +0000 UTC m=+0.343637899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109075 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fb86fcbd27ce842402ef800352681 SystemUUID:ec2fb86f-cbd2-7ce8-4240-2ef800352681 BootID:5527dc6b-d434-4c49-baf2-36e2e7458972 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:c6:86:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:c6:86:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:f8:28:ce:02:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 17:12:03.203856 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.203844    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 17:12:03.204057 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.204035    2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 17:12:03.206952 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.206924    2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 17:12:03.207108 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.206955    2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 17:12:03.207152 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.207111    2570 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 17:12:03.207152 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.207119    2570 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 17:12:03.207152 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.207133    2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 17:12:03.207152 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.207143    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 17:12:03.208010 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.207999    2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 17:12:03.211701 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.211687    2570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 17:12:03.211834 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.211824    2570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 17:12:03.212558 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.212543    2570 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 17:12:03.212593 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.212566    2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 17:12:03.212593 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.212582    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 17:12:03.212682 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.212596    2570 kubelet.go:405] "Adding apiserver pod source"
Oct 30 17:12:03.212682 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.212609    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 17:12:03.213139 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.213115    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 17:12:03.215369 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.215352    2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 17:12:03.222096 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.222072    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 17:12:03.222837 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.222825    2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 17:12:03.222877 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.222856    2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 17:12:03.223035 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223025    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 17:12:03.223084 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223041    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 17:12:03.223084 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223047    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 17:12:03.223084 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223053    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 17:12:03.223084 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223059    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 17:12:03.223084 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223065    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 17:12:03.223675 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223665    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 17:12:03.223675 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223674    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 17:12:03.223735 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223682    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 17:12:03.223735 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223688    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 17:12:03.223735 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223697    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 17:12:03.223735 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.223707    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 17:12:03.224432 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.224422    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 17:12:03.224432 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.224432    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 17:12:03.228285 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.228271    2570 server.go:1263] "Started kubelet"
Oct 30 17:12:03.228370 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.228342    2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 17:12:03.228464 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.228398    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 17:12:03.228565 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.228508    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 17:12:03.229174 ip-10-0-135-170 systemd[1]: Started Kubernetes Kubelet.
Oct 30 17:12:03.229822 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.229742    2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 17:12:03.230344 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.230328    2570 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 17:12:03.233219 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.233203    2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 17:12:03.235186 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.235171    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 17:12:03.235706 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.235693    2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 17:12:03.235890 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.235842    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-170.ec2.internal\" not found"
Oct 30 17:12:03.236360 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.236344    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 17:12:03.236360 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.236348    2570 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 17:12:03.236486 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.236370    2570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 17:12:03.236486 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.236482    2570 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 17:12:03.236580 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.236494    2570 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 17:12:03.240182 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240167    2570 factory.go:55] Registering systemd factory
Oct 30 17:12:03.240182 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240183    2570 factory.go:223] Registration of the systemd container factory successfully
Oct 30 17:12:03.240453 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240438    2570 factory.go:153] Registering CRI-O factory
Oct 30 17:12:03.240499 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240457    2570 factory.go:223] Registration of the crio container factory successfully
Oct 30 17:12:03.240543 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240501    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 17:12:03.240543 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240521    2570 factory.go:103] Registering Raw factory
Oct 30 17:12:03.240543 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.240531    2570 manager.go:1199] Started watching for new ooms in manager
Oct 30 17:12:03.241441 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.241423    2570 manager.go:319] Starting recovery of all containers
Oct 30 17:12:03.253485 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.253459    2570 manager.go:324] Recovery completed
Oct 30 17:12:03.255483 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.255447    2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 17:12:03.259128 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.259110    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 17:12:03.260977 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.260949    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 17:12:03.261041 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.260992    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 17:12:03.261041 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.261007    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientPID"
Oct 30 17:12:03.261681 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.261630    2570 cpu_manager.go:226] "Starting" policy="none"
Oct 30 17:12:03.261681 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.261665    2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 17:12:03.261837 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.261686    2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 17:12:03.261837 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.261636    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 17:12:03.262488 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.262473    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 17:12:03.262555 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.262495    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 17:12:03.262555 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.262505    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientPID"
Oct 30 17:12:03.262629 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.262601    2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 17:12:03.262713 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.262613    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 17:12:03.263610 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.262606    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-170.ec2.internal.18e35de6f931e93d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-170.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-170.ec2.internal,},FirstTimestamp:2026-10-30 17:12:03.228248381 +0000 UTC m=+0.370366350,LastTimestamp:2026-10-30 17:12:03.228248381 +0000 UTC m=+0.370366350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-170.ec2.internal,}"
Oct 30 17:12:03.263845 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.263833    2570 policy_none.go:50] "Start"
Oct 30 17:12:03.263881 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.263850    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 17:12:03.263881 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.263860    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 17:12:03.265726 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.265711    2570 policy_none.go:44] "Start"
Oct 30 17:12:03.266915 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.266899    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 17:12:03.266978 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.266967    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bz2fj"
Oct 30 17:12:03.274860 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.274691    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bz2fj"
Oct 30 17:12:03.278434 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.278408    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 17:12:03.278683 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.278657    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 17:12:03.298372 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.298330    2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 17:12:03.300977 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.300964    2570 manager.go:341] "Starting Device Plugin manager"
Oct 30 17:12:03.301665 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.301636    2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 17:12:03.301665 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.301664    2570 server.go:87] "Starting device plugin registration server"
Oct 30 17:12:03.301916 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.301906    2570 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 17:12:03.301970 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.301922    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 17:12:03.302108 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.302086    2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 17:12:03.302560 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.302225    2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 17:12:03.302560 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.302242    2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 17:12:03.302710 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.302623    2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 17:12:03.302710 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.302684    2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-170.ec2.internal\" not found"
Oct 30 17:12:03.313980 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.313957    2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:12:03.385774 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.385730    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 17:12:03.387704 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.387055    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 17:12:03.388071 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.388055    2570 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 17:12:03.388127 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.388095    2570 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 17:12:03.388176 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:03.388137    2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 17:12:03.402403 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.402325    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 17:12:03.403037 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.403016    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:12:03.403362 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.403342    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 17:12:03.403462 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.403380    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 17:12:03.403462 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.403397    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientPID"
Oct 30 17:12:03.403462 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.403429    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.424022 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.423993    2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.488903 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.488859    2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"]
Oct 30 17:12:03.491227 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.491203    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.492876 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.492858    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.522233 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.522211    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.524893 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.524879    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.537622 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.537599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.537715 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.537637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.537715 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.537678    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd29d125f430f9e0a8a7b7b6580a8dec-config\") pod \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" (UID: \"cd29d125f430f9e0a8a7b7b6580a8dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.547588 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.547567    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 17:12:03.547588 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.547585    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 17:12:03.637976 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.637929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.637976 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.637982    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.638177 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.638007    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd29d125f430f9e0a8a7b7b6580a8dec-config\") pod \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" (UID: \"cd29d125f430f9e0a8a7b7b6580a8dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.638177 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.638023    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.638177 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.638035    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd29d125f430f9e0a8a7b7b6580a8dec-config\") pod \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" (UID: \"cd29d125f430f9e0a8a7b7b6580a8dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.638177 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.638049    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.852188 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.852149    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:03.854331 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:03.854314    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:12:04.171765 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.171687    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 17:12:04.172250 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.171913    2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 17:12:04.213657 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.213604    2570 apiserver.go:52] "Watching apiserver"
Oct 30 17:12:04.235492 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.235466    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 17:12:04.262661 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.262620    2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 17:12:04.262964 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.262938    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7k5d2","openshift-multus/multus-additional-cni-plugins-5xbwb","openshift-network-diagnostics/network-check-target-z2hmf","openshift-dns/node-resolver-zz6f9","openshift-network-operator/iptables-alerter-6nccg","kube-system/global-pull-secret-syncer-v8rgj","kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal","openshift-image-registry/node-ca-xjh5h","kube-system/konnectivity-agent-4hzkr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal","openshift-multus/network-metrics-daemon-dqkvr","openshift-ovn-kubernetes/ovnkube-node-96dfd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw","openshift-cluster-node-tuning-operator/tuned-nc4z8"]
Oct 30 17:12:04.265879 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.265860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.265967 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.265947    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:04.268155 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.268137    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.270739 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.270717    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.273461 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.273440    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.276092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.276074    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.276253 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.276124    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:04.279115 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.279077    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:07:03 +0000 UTC" deadline="2028-05-24 11:51:52.122663 +0000 UTC"
Oct 30 17:12:04.279115 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.279112    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13722h39m47.843553519s"
Oct 30 17:12:04.279260 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.279186    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.281566 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.281302    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.285180 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.285159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.286269 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.286136    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.286371 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.286308    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.286434 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.286421    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.286719 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.286654    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.286719 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.286690    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn7sb\""
Oct 30 17:12:04.286851 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.286777    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkzvv\""
Oct 30 17:12:04.288696 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.287635    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:04.288696 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.287720    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:04.288933 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.288906    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 17:12:04.288933 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.288922    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g55vh\""
Oct 30 17:12:04.289090 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.288973    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.289090 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.288998    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.290453 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.290436    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.292349 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.292331    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.293194 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.293171    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 17:12:04.293286 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.293235    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zbs9x"
Oct 30 17:12:04.294553 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.294533    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.340495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.339076    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 17:12:04.342461 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342435    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-etc-tuned\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.342558 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.342558 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342484    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-cni-binary-copy\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.342558 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-systemd-units\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.342691 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-var-lib-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.342691 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-bin\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.342691 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342634    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcdl\" (UniqueName: \"kubernetes.io/projected/84532c65-2d04-45a5-b0cb-9841d8e20d87-kube-api-access-mlcdl\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.342777 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.342777 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5lhx\" (UniqueName: \"kubernetes.io/projected/412abd52-f4da-491a-b694-30750614e3b4-kube-api-access-x5lhx\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.342777 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-env-overrides\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.342777 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc79p\" (UniqueName: \"kubernetes.io/projected/34ccfea3-7a41-4925-9669-c4a25efabc4a-kube-api-access-hc79p\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.342890 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342798    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d255144b-ecb7-489b-98ca-a188acb6c109-konnectivity-ca\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.342890 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-conf\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.342890 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-sys\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.342890 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342887    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-tmp\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.343003 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342905    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-kubelet\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343003 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltk9\" (UniqueName: \"kubernetes.io/projected/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-kube-api-access-mltk9\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.343003 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342955    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.343003 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.342972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a3c7100-3ff5-4410-abd7-03e68c27baa3-host\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.343118 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/412abd52-f4da-491a-b694-30750614e3b4-host-slash\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.343118 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-multus-daemon-config\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.343118 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343061    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-slash\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343118 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-netd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343241 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343128    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.343241 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343158    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:04.343241 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-host\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.343241 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343205    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-conf-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.343241 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343220    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-device-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.343376 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343244    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mzl9\" (UniqueName: \"kubernetes.io/projected/622e2edd-70ee-4176-8498-50a740a59f82-kube-api-access-4mzl9\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.343376 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zv5\" (UniqueName: \"kubernetes.io/projected/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kube-api-access-j6zv5\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.343376 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343306    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d255144b-ecb7-489b-98ca-a188acb6c109-agent-certs\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.343376 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343336    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-modprobe-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.343376 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-socket-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.343526 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-kubernetes\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.343526 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-k8s-cni-cncf-io\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.343526 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-kubelet-config\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.343526 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343473    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a3c7100-3ff5-4410-abd7-03e68c27baa3-serviceca\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.343526 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-socket-dir-parent\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.343689 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343549    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-ovn\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343689 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343575    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwlt\" (UniqueName: \"kubernetes.io/projected/7a3c7100-3ff5-4410-abd7-03e68c27baa3-kube-api-access-sxwlt\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.343689 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343590    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.343689 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-run\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.343689 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343674    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-hostroot\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.343826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343709    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-netns\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-node-log\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-config\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.343826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343775    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.343826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpshs\" (UniqueName: \"kubernetes.io/projected/991f66af-6bcb-4bd8-921f-58f2fb879067-kube-api-access-fpshs\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.343826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343813    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysconfig\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343835    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-systemd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343855    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-multus-certs\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343881    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-systemd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343918    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343942    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-registration-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343975    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-var-lib-kubelet\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.343988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-multus\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-etc-kubernetes\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344069    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-etc-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-script-lib\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-sys-fs\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddxd\" (UniqueName: \"kubernetes.io/projected/caa54cca-be01-4afd-8690-5cb0958c1027-kube-api-access-5ddxd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344175    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-cnibin\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344251    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-cnibin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344303 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344302    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-bin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344326    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-log-socket\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344348    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/991f66af-6bcb-4bd8-921f-58f2fb879067-hosts-file\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344362    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-dbus\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344403    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-lib-modules\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-system-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-netns\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344479    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-kubelet\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344497    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344516    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovn-node-metrics-cert\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/412abd52-f4da-491a-b694-30750614e3b4-iptables-alerter-script\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.344592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-system-cni-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.344897 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.344608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f66af-6bcb-4bd8-921f-58f2fb879067-tmp-dir\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.386587 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386557    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386591    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386694    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386716    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386726    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386733    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386773    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.386800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386800    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386831    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386852    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vxcg\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386854    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386876    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2wbm5\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386884    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386903    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386914    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctbv7\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386925    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386934    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386904    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386948    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386954    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8szd\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.386979    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.387008    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49f6v\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.387014    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.387039    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 17:12:04.387129 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.387045    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lrx5\""
Oct 30 17:12:04.387577 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.387141    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 17:12:04.428318 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.428254    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode477122cbf7bd238eb683471d99d303b.slice/crio-84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43 WatchSource:0}: Error finding container 84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43: Status 404 returned error can't find the container with id 84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43
Oct 30 17:12:04.428622 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.428596    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd29d125f430f9e0a8a7b7b6580a8dec.slice/crio-b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee WatchSource:0}: Error finding container b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee: Status 404 returned error can't find the container with id b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee
Oct 30 17:12:04.434493 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.434479    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:12:04.445444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445402    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:04.445532 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445455    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-host\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.445578 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445533    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-host\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.445657 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445624    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-conf-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.445722 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-device-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.445722 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445697    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-conf-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.445825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzl9\" (UniqueName: \"kubernetes.io/projected/622e2edd-70ee-4176-8498-50a740a59f82-kube-api-access-4mzl9\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.445825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445742    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-device-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.445825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zv5\" (UniqueName: \"kubernetes.io/projected/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kube-api-access-j6zv5\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.445825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445774    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d255144b-ecb7-489b-98ca-a188acb6c109-agent-certs\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.445825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-modprobe-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-socket-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445879    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-kubernetes\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445925    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-k8s-cni-cncf-io\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-kubelet-config\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445954    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-modprobe-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445960    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-socket-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445982    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-kubernetes\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446002    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-k8s-cni-cncf-io\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.445980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a3c7100-3ff5-4410-abd7-03e68c27baa3-serviceca\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-socket-dir-parent\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446046    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-kubelet-config\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446061    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-ovn\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwlt\" (UniqueName: \"kubernetes.io/projected/7a3c7100-3ff5-4410-abd7-03e68c27baa3-kube-api-access-sxwlt\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446105    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-ovn\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446119    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446125    2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-run\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446162    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-hostroot\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446167    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-socket-dir-parent\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446184    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-netns\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446226    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-netns\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446221    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-hostroot\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446248    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-run\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446260    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-node-log\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446331    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-config\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446355    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446375    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-node-log\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446380    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpshs\" (UniqueName: \"kubernetes.io/projected/991f66af-6bcb-4bd8-921f-58f2fb879067-kube-api-access-fpshs\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446412    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a3c7100-3ff5-4410-abd7-03e68c27baa3-serviceca\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.446754 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446428    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysconfig\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446456    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-systemd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446491    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-systemd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446515    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-multus-certs\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446539    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysconfig\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446549    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-systemd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446570    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-multus-certs\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446605    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446613    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-systemd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446667    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446609    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-registration-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-var-lib-kubelet\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446742    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446766    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-multus\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446770    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-registration-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.447594 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446801    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-etc-kubernetes\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446825    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-etc-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446848    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-script-lib\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446870    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-sys-fs\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446842    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-var-lib-kubelet\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddxd\" (UniqueName: \"kubernetes.io/projected/caa54cca-be01-4afd-8690-5cb0958c1027-kube-api-access-5ddxd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446905    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-etc-kubernetes\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446946    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-etc-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446959    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-cnibin\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446982    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446992    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447030    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447056    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-cnibin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-bin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447102    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-log-socket\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447124    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/991f66af-6bcb-4bd8-921f-58f2fb879067-hosts-file\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-dbus\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.448624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446907    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-lib-modules\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-system-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447299    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-lib-modules\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-netns\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447326    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-kubelet\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447340    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-netns\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447376    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovn-node-metrics-cert\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447384    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-multus\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/412abd52-f4da-491a-b694-30750614e3b4-iptables-alerter-script\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447415    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-system-cni-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447436    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-system-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447239    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-cnibin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447440    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f66af-6bcb-4bd8-921f-58f2fb879067-tmp-dir\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-etc-tuned\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447520    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447530    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-config\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.449495 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.447542    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447546    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-cni-binary-copy\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447570    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-systemd-units\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447582    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-log-socket\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447565    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.447618    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:04.947580013 +0000 UTC m=+2.089697989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447626    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-bin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.446963    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-sys-fs\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447684    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/991f66af-6bcb-4bd8-921f-58f2fb879067-hosts-file\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447704    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-systemd-units\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447751    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-dbus\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447776    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-cnibin\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-var-lib-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447901    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-bin\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447912    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-script-lib\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcdl\" (UniqueName: \"kubernetes.io/projected/84532c65-2d04-45a5-b0cb-9841d8e20d87-kube-api-access-mlcdl\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447935    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-kubelet\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.450482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.447965    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-var-lib-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lhx\" (UniqueName: \"kubernetes.io/projected/412abd52-f4da-491a-b694-30750614e3b4-kube-api-access-x5lhx\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448010    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-bin\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-env-overrides\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448047    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448073    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc79p\" (UniqueName: \"kubernetes.io/projected/34ccfea3-7a41-4925-9669-c4a25efabc4a-kube-api-access-hc79p\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448098    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d255144b-ecb7-489b-98ca-a188acb6c109-konnectivity-ca\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448122    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-conf\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-sys\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-tmp\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448191    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-kubelet\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448218    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltk9\" (UniqueName: \"kubernetes.io/projected/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-kube-api-access-mltk9\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448267    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a3c7100-3ff5-4410-abd7-03e68c27baa3-host\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448268    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-sys\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448166    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/412abd52-f4da-491a-b694-30750614e3b4-iptables-alerter-script\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.451273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448341    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-kubelet\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448353    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.448581    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.448626    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:04.948610938 +0000 UTC m=+2.090728894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448726    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a3c7100-3ff5-4410-abd7-03e68c27baa3-host\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448749    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-env-overrides\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448794    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d255144b-ecb7-489b-98ca-a188acb6c109-konnectivity-ca\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448872    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-conf\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448919    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-system-cni-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448949    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/412abd52-f4da-491a-b694-30750614e3b4-host-slash\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448963    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.448991    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-multus-daemon-config\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449016    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-slash\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449026    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/412abd52-f4da-491a-b694-30750614e3b4-host-slash\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-netd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449073    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449554    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-slash\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452011 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449608    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-netd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452763 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.449841    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-cni-binary-copy\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.452763 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.450409    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-etc-tuned\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.452763 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.450430    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovn-node-metrics-cert\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.452763 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.450617    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d255144b-ecb7-489b-98ca-a188acb6c109-agent-certs\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.452763 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.451053    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-tmp\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.453542 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.453523    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.453658 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.453625    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f66af-6bcb-4bd8-921f-58f2fb879067-tmp-dir\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.453719 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.453707    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.453772 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.453721    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-multus-daemon-config\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.467365 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.467344    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:04.467365 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.467365    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:04.467464 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.467375    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:04.467464 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.467421    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:12:04.967408568 +0000 UTC m=+2.109526548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:04.469849 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.469823    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zv5\" (UniqueName: \"kubernetes.io/projected/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kube-api-access-j6zv5\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.470578 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.470555    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzl9\" (UniqueName: \"kubernetes.io/projected/622e2edd-70ee-4176-8498-50a740a59f82-kube-api-access-4mzl9\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.471233 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.471210    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpshs\" (UniqueName: \"kubernetes.io/projected/991f66af-6bcb-4bd8-921f-58f2fb879067-kube-api-access-fpshs\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.471500 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.471436    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddxd\" (UniqueName: \"kubernetes.io/projected/caa54cca-be01-4afd-8690-5cb0958c1027-kube-api-access-5ddxd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.471500 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.471439    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwlt\" (UniqueName: \"kubernetes.io/projected/7a3c7100-3ff5-4410-abd7-03e68c27baa3-kube-api-access-sxwlt\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.473077 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.473050    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lhx\" (UniqueName: \"kubernetes.io/projected/412abd52-f4da-491a-b694-30750614e3b4-kube-api-access-x5lhx\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.473167 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.473134    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltk9\" (UniqueName: \"kubernetes.io/projected/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-kube-api-access-mltk9\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.473167 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.473142    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcdl\" (UniqueName: \"kubernetes.io/projected/84532c65-2d04-45a5-b0cb-9841d8e20d87-kube-api-access-mlcdl\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.473393 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.473376    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc79p\" (UniqueName: \"kubernetes.io/projected/34ccfea3-7a41-4925-9669-c4a25efabc4a-kube-api-access-hc79p\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.599820 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.599788    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:12:04.610333 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.610310    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa54cca_be01_4afd_8690_5cb0958c1027.slice/crio-5066ca29abdb32c6a3b225a7adb3bee25ad47f8a4624304b212a51552e4ca7bb WatchSource:0}: Error finding container 5066ca29abdb32c6a3b225a7adb3bee25ad47f8a4624304b212a51552e4ca7bb: Status 404 returned error can't find the container with id 5066ca29abdb32c6a3b225a7adb3bee25ad47f8a4624304b212a51552e4ca7bb
Oct 30 17:12:04.615691 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.615673    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:12:04.624167 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.624144    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991f66af_6bcb_4bd8_921f_58f2fb879067.slice/crio-f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2 WatchSource:0}: Error finding container f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2: Status 404 returned error can't find the container with id f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2
Oct 30 17:12:04.637705 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.637685    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:12:04.645692 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.645670    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412abd52_f4da_491a_b694_30750614e3b4.slice/crio-bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82 WatchSource:0}: Error finding container bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82: Status 404 returned error can't find the container with id bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82
Oct 30 17:12:04.647763 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.647746    2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:12:04.651050 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.651035    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7k5d2"
Oct 30 17:12:04.659535 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.659500    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622e2edd_70ee_4176_8498_50a740a59f82.slice/crio-ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a WatchSource:0}: Error finding container ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a: Status 404 returned error can't find the container with id ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a
Oct 30 17:12:04.673131 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.673115    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:12:04.681466 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.681448    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3c7100_3ff5_4410_abd7_03e68c27baa3.slice/crio-3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b WatchSource:0}: Error finding container 3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b: Status 404 returned error can't find the container with id 3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b
Oct 30 17:12:04.691691 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.691674    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:04.714870 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.714852    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:04.723791 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.723761    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84532c65_2d04_45a5_b0cb_9841d8e20d87.slice/crio-c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf WatchSource:0}: Error finding container c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf: Status 404 returned error can't find the container with id c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf
Oct 30 17:12:04.730986 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.730967    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:12:04.739999 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.739978    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1c6d38_094f_41a0_8c77_b1f54d231ea7.slice/crio-5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d WatchSource:0}: Error finding container 5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d: Status 404 returned error can't find the container with id 5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d
Oct 30 17:12:04.746970 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.746951    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:12:04.757691 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:04.757666    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ccfea3_7a41_4925_9669_c4a25efabc4a.slice/crio-b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38 WatchSource:0}: Error finding container b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38: Status 404 returned error can't find the container with id b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38
Oct 30 17:12:04.835435 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.835405    2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:12:04.952831 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.952749    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:04.952993 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:04.952841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:04.952993 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.952897    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:04.952993 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.952972    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:04.953147 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.952975    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:05.952957209 +0000 UTC m=+3.095075180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:04.953147 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:04.953027    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:05.953002796 +0000 UTC m=+3.095120754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:05.053259 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.053221    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:05.053488 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.053443    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:05.053488 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.053467    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:05.053488 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.053482    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:05.053666 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.053548    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:12:06.053528467 +0000 UTC m=+3.195646433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:05.146401 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.146344    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:12:05.296920 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.296816    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:07:04 +0000 UTC" deadline="2028-06-28 18:57:02.655600938 +0000 UTC"
Oct 30 17:12:05.296920 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.296871    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14569h44m57.358733682s"
Oct 30 17:12:05.418307 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.418237    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf"}
Oct 30 17:12:05.427087 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.427046    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hzkr" event={"ID":"d255144b-ecb7-489b-98ca-a188acb6c109","Type":"ContainerStarted","Data":"ee8cc6acf6a43ba59903ba132a699792b44857324f40679998426734d7c6ba89"}
Oct 30 17:12:05.432511 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.432424    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xjh5h" event={"ID":"7a3c7100-3ff5-4410-abd7-03e68c27baa3","Type":"ContainerStarted","Data":"3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b"}
Oct 30 17:12:05.444551 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.444466    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7k5d2" event={"ID":"622e2edd-70ee-4176-8498-50a740a59f82","Type":"ContainerStarted","Data":"ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a"}
Oct 30 17:12:05.449147 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.449074    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6nccg" event={"ID":"412abd52-f4da-491a-b694-30750614e3b4","Type":"ContainerStarted","Data":"bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82"}
Oct 30 17:12:05.455333 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.455256    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zz6f9" event={"ID":"991f66af-6bcb-4bd8-921f-58f2fb879067","Type":"ContainerStarted","Data":"f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2"}
Oct 30 17:12:05.466764 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.466681    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nc4z8" event={"ID":"caa54cca-be01-4afd-8690-5cb0958c1027","Type":"ContainerStarted","Data":"5066ca29abdb32c6a3b225a7adb3bee25ad47f8a4624304b212a51552e4ca7bb"}
Oct 30 17:12:05.484716 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.484637    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" event={"ID":"cd29d125f430f9e0a8a7b7b6580a8dec","Type":"ContainerStarted","Data":"b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee"}
Oct 30 17:12:05.495952 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.495920    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerStarted","Data":"b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38"}
Oct 30 17:12:05.509966 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.508000    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw" event={"ID":"6a1c6d38-094f-41a0-8c77-b1f54d231ea7","Type":"ContainerStarted","Data":"5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d"}
Oct 30 17:12:05.510117 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.510035    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" event={"ID":"e477122cbf7bd238eb683471d99d303b","Type":"ContainerStarted","Data":"84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43"}
Oct 30 17:12:05.959936 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.959904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:05.960137 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:05.959952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:05.960137 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.960075    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:05.960137 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.960137    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:07.960117721 +0000 UTC m=+5.102235682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:05.960315 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.960211    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:05.960315 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:05.960243    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:07.960232526 +0000 UTC m=+5.102350501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:06.060817 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:06.060692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:06.061006 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.060892    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:06.061006 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.060912    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:06.061006 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.060925    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:06.061006 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.060978    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:12:08.060960416 +0000 UTC m=+5.203078390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:06.297836 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:06.297723    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:07:04 +0000 UTC" deadline="2028-04-19 06:13:12.379758072 +0000 UTC"
Oct 30 17:12:06.297836 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:06.297773    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12877h1m6.081989882s"
Oct 30 17:12:06.389200 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:06.389029    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:06.389200 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.389179    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:06.389738 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:06.389598    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:06.389738 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.389723    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:06.389837 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:06.389802    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:06.389923 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:06.389875    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:07.978610 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:07.978572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:07.979108 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:07.978639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:07.979108 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:07.978801    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:07.979108 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:07.978875    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.978856232 +0000 UTC m=+9.120974196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:07.979598 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:07.979574    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:07.979697 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:07.979640    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.979622018 +0000 UTC m=+9.121739977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:08.079195 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.079101    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:08.079378 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.079271    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:08.079378 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.079293    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:08.079378 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.079307    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:08.079378 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.079370    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.079352087 +0000 UTC m=+9.221470061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:08.389046 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.389014    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:08.389192 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.389142    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:08.389484 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.389467    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:08.389561 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.389542    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:08.389618 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.389582    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:08.389679 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:08.389626    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:08.542503 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.542457    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerStarted","Data":"09b32b1fcdfbe4c55fd9afa3c7c4e924e6c15bf6aa60c1cfab3cc08a5e7f8ec9"}
Oct 30 17:12:08.547641 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.547604    2570 generic.go:356] "Generic (PLEG): container finished" podID="e477122cbf7bd238eb683471d99d303b" containerID="1982b1ac3d580ba1bc1ba397561f5bde56e2f1b61a25953809694dcb9fa11c80" exitCode=0
Oct 30 17:12:08.547825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:08.547676    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" event={"ID":"e477122cbf7bd238eb683471d99d303b","Type":"ContainerDied","Data":"1982b1ac3d580ba1bc1ba397561f5bde56e2f1b61a25953809694dcb9fa11c80"}
Oct 30 17:12:09.557989 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:09.557457    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" event={"ID":"e477122cbf7bd238eb683471d99d303b","Type":"ContainerStarted","Data":"1d85f8985251b8c3667d484acb78c7699ab3ae530ac0037c40744095ff0a97fb"}
Oct 30 17:12:09.561852 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:09.561818    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" event={"ID":"cd29d125f430f9e0a8a7b7b6580a8dec","Type":"ContainerStarted","Data":"9b60889084568a91f326fabbc49d493365b0f617afe55d460797520807b23e4f"}
Oct 30 17:12:09.567261 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:09.566553    2570 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="09b32b1fcdfbe4c55fd9afa3c7c4e924e6c15bf6aa60c1cfab3cc08a5e7f8ec9" exitCode=0
Oct 30 17:12:09.567261 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:09.566612    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerDied","Data":"09b32b1fcdfbe4c55fd9afa3c7c4e924e6c15bf6aa60c1cfab3cc08a5e7f8ec9"}
Oct 30 17:12:09.659811 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:09.658406    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" podStartSLOduration=5.65838347 podStartE2EDuration="5.65838347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:09.608255572 +0000 UTC m=+6.750373556" watchObservedRunningTime="2026-10-30 17:12:09.65838347 +0000 UTC m=+6.800501459"
Oct 30 17:12:10.388964 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:10.388924    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:10.389161 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:10.388937    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:10.389161 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:10.389075    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:10.389161 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:10.389077    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:10.389337 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:10.389163    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:10.389337 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:10.389234    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:10.578330 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:10.577782    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hzkr" event={"ID":"d255144b-ecb7-489b-98ca-a188acb6c109","Type":"ContainerStarted","Data":"378126cafba1cccfa2f6be0f4a70c0d5506f55cd4d031c00a2cb060e687fc02d"}
Oct 30 17:12:10.623401 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:10.623035    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" podStartSLOduration=6.623008474 podStartE2EDuration="6.623008474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:09.702408398 +0000 UTC m=+6.844526400" watchObservedRunningTime="2026-10-30 17:12:10.623008474 +0000 UTC m=+7.765126456"
Oct 30 17:12:11.105073 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:11.105035    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:11.283404 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:11.283213    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:11.586723 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:11.586672    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw" event={"ID":"6a1c6d38-094f-41a0-8c77-b1f54d231ea7","Type":"ContainerStarted","Data":"a96cb26edfc4626c60a2a0371664f98ff586895d823b91358e0e2bfb3a71405d"}
Oct 30 17:12:11.590738 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:11.590710    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xjh5h" event={"ID":"7a3c7100-3ff5-4410-abd7-03e68c27baa3","Type":"ContainerStarted","Data":"2e0de6bac9b04eb988e1d4e940d01887507b441a49b6a81efafa9e2911fdfecf"}
Oct 30 17:12:11.634254 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:11.634201    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4hzkr" podStartSLOduration=2.787875819 podStartE2EDuration="7.634185087s" totalImagesPullingTime="4.846309268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:10.623372644 +0000 UTC m=+7.765490630" watchObservedRunningTime="2026-10-30 17:12:11.634185087 +0000 UTC m=+8.776303069"
Oct 30 17:12:12.012290 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.011413    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:12.012290 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.011484    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:12.012290 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.011652    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:12.012290 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.011747    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:20.011723332 +0000 UTC m=+17.153841522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:12.012290 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.012170    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:12.012290 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.012222    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:20.012206606 +0000 UTC m=+17.154324567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:12.084252 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.084158    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:12.112858 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.112812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:12.113039 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.113028    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:12.113100 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.113049    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:12.113100 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.113063    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:12.113211 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.113110    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:12:20.113096176 +0000 UTC m=+17.255214135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:12.254268 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.254170    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:12.382655 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.382604    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:12.389273 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.389244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:12.389407 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.389381    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:12.389851 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.389832    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:12.389978 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.389958    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:12.390030 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.390019    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:12.390109 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:12.390093    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:12.499073 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.499002    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:12.613793 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:12.613757    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:13.081113 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.081063    2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 17:12:13.308527 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.308419    2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:12:13.08108757Z","UUID":"fcd847b3-1682-4253-8356-762a6de40a19","Handler":null,"Name":"","Endpoint":""}
Oct 30 17:12:13.310480 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.310453    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 17:12:13.310610 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.310509    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 17:12:13.432433 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.432356    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:13.433156 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.433131    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:13.475512 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.475463    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xjh5h" podStartSLOduration=3.476747022 podStartE2EDuration="9.475445665s" totalImagesPullingTime="5.998698643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:11.634842648 +0000 UTC m=+8.776960637" watchObservedRunningTime="2026-10-30 17:12:13.475445665 +0000 UTC m=+10.617563645"
Oct 30 17:12:13.596865 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.596830    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7k5d2" event={"ID":"622e2edd-70ee-4176-8498-50a740a59f82","Type":"ContainerStarted","Data":"8a3b4db32e16aaed092b533c68e1eca620c5c13c3ae1ee9016a066337923a3e5"}
Oct 30 17:12:13.603765 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:13.603741    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw" event={"ID":"6a1c6d38-094f-41a0-8c77-b1f54d231ea7","Type":"ContainerStarted","Data":"a843f32684fa6ee763de4920fe467b7357377da0e41be885ff6d69787ad5e192"}
Oct 30 17:12:14.115459 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.115412    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:14.222894 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.222685    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:14.388785 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.388754    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:14.388938 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:14.388882    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:14.389241 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.389226    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:14.389328 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:14.389310    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:14.389376 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.389364    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:14.389442 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:14.389429    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:14.606807 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.606765    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zz6f9" event={"ID":"991f66af-6bcb-4bd8-921f-58f2fb879067","Type":"ContainerStarted","Data":"949760caacf19b12245137c8ad41adcd578245c7587bec91984ab02be33f7e5c"}
Oct 30 17:12:14.608802 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.608772    2570 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="b42b2114e954bcd0c8a0183b56d7c1caa7c788ef4d8d52a245a2f7496157451b" exitCode=0
Oct 30 17:12:14.608917 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.608851    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerDied","Data":"b42b2114e954bcd0c8a0183b56d7c1caa7c788ef4d8d52a245a2f7496157451b"}
Oct 30 17:12:14.612299 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.612270    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw" event={"ID":"6a1c6d38-094f-41a0-8c77-b1f54d231ea7","Type":"ContainerStarted","Data":"9f4aa8a57633438878cd9a08050389ffd5819988c1dede5cfa9531616ffa76e1"}
Oct 30 17:12:14.647742 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.647685    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7k5d2" podStartSLOduration=2.18788277 podStartE2EDuration="10.647667649s" totalImagesPullingTime="8.459784879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:13.641793605 +0000 UTC m=+10.783911584" watchObservedRunningTime="2026-10-30 17:12:14.647667649 +0000 UTC m=+11.789785624"
Oct 30 17:12:14.648076 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.648038    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zz6f9" podStartSLOduration=1.8180444119999999 podStartE2EDuration="10.648025194s" totalImagesPullingTime="8.829980782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:14.64740383 +0000 UTC m=+11.789521820" watchObservedRunningTime="2026-10-30 17:12:14.648025194 +0000 UTC m=+11.790143175"
Oct 30 17:12:14.739375 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:14.739321    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw" podStartSLOduration=1.809317578 podStartE2EDuration="10.739308644s" totalImagesPullingTime="8.929991066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:14.738890561 +0000 UTC m=+11.881008542" watchObservedRunningTime="2026-10-30 17:12:14.739308644 +0000 UTC m=+11.881426622"
Oct 30 17:12:15.523439 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.523364    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:15.523898 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.523509    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 17:12:15.523984 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.523943    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:12:15.617053 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.617018    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6nccg" event={"ID":"412abd52-f4da-491a-b694-30750614e3b4","Type":"ContainerStarted","Data":"6904a94169f2b73bb99902db8f945bafb4b22b167ec3a445e178afb32f1bcbaf"}
Oct 30 17:12:15.619618 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.619577    2570 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="7683259cfd3132ad3c313310dbdf7114c0d51786eebd25197d82695ce98e11b5" exitCode=0
Oct 30 17:12:15.619771 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.619633    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerDied","Data":"7683259cfd3132ad3c313310dbdf7114c0d51786eebd25197d82695ce98e11b5"}
Oct 30 17:12:15.643247 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.643213    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:15.658994 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:15.658949    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6nccg" podStartSLOduration=2.694479884 podStartE2EDuration="11.658930698s" totalImagesPullingTime="8.964450814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:15.658833914 +0000 UTC m=+12.800951894" watchObservedRunningTime="2026-10-30 17:12:15.658930698 +0000 UTC m=+12.801048678"
Oct 30 17:12:16.388688 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.388636    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:16.388688 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.388691    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:16.388925 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.388766    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:16.388925 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:16.388781    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:16.388925 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:16.388880    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:16.389079 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:16.388978    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:16.624073 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.624029    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nc4z8" event={"ID":"caa54cca-be01-4afd-8690-5cb0958c1027","Type":"ContainerStarted","Data":"e334b2a31e20ecc06cb533ca4aca767999b449672cf81a89abb34764baa1baf2"}
Oct 30 17:12:16.626826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.626800    2570 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="b1f61f3376b8693f2014426681afdda90e777fb24750d53245b11eb9ead48ca8" exitCode=0
Oct 30 17:12:16.626956 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.626886    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerDied","Data":"b1f61f3376b8693f2014426681afdda90e777fb24750d53245b11eb9ead48ca8"}
Oct 30 17:12:16.668382 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:16.668283    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nc4z8" podStartSLOduration=2.743309417 podStartE2EDuration="13.668269721s" totalImagesPullingTime="10.924960304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:16.667973445 +0000 UTC m=+13.810091442" watchObservedRunningTime="2026-10-30 17:12:16.668269721 +0000 UTC m=+13.810387699"
Oct 30 17:12:17.403960 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:17.403920    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:17.515435 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:17.515392    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:17.686696 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:17.686590    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:17.797693 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:17.797629    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:18.388675 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.388624    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:18.388842 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:18.388764    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:18.388842 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.388813    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:18.388950 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:18.388938    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:18.389005 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.388982    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:18.389082 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:18.389062    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:18.635587 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.635329    2570 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="841b718a9a3562f7985f8c7167653a675d00eea9a8d444711dd72e93b6620bc6" exitCode=0
Oct 30 17:12:18.635587 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.635396    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerDied","Data":"841b718a9a3562f7985f8c7167653a675d00eea9a8d444711dd72e93b6620bc6"}
Oct 30 17:12:18.707065 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.707028    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:18.905715 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:18.905579    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:19.277417 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:19.277325    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:19.641404 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:19.641370    2570 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="475670f469f57f31dacb73ce4ba279d122db8f93dc9a89a3a1351dcf33e41a26" exitCode=0
Oct 30 17:12:19.641565 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:19.641424    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerDied","Data":"475670f469f57f31dacb73ce4ba279d122db8f93dc9a89a3a1351dcf33e41a26"}
Oct 30 17:12:19.858187 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:19.858157    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:19.973554 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:19.973472    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:20.074738 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.074704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:20.074886 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.074752    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:20.074886 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.074859    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:20.074886 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.074864    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:20.074975 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.074913    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:36.07490122 +0000 UTC m=+33.217019177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:20.074975 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.074927    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:36.074920619 +0000 UTC m=+33.217038576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:20.126779 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.126746    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:20.175988 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.175952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:20.176118 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.176090    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:20.176118 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.176105    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:20.176118 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.176114    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:20.176200 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.176167    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:12:36.176149945 +0000 UTC m=+33.318267920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:20.234092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.234018    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:20.257461 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.257433    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zz6f9_991f66af-6bcb-4bd8-921f-58f2fb879067/dns-node-resolver/0.log"
Oct 30 17:12:20.345374 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.345334    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:20.388760 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.388738    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:20.388867 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.388766    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:20.388867 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.388826    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:20.388951 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.388874    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:20.388999 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.388958    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:20.389076 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:20.389062    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:20.492229 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.492173    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:20.645856 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.645831    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"c87342cad7f7f538fa751921ff9b6e780f266a4ee17bd68d0a248c9bc7a7b107"}
Oct 30 17:12:20.645949 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.645894    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"9abd58dfe42bf55c1e913507ff4f6399b50596e49d70ba4f71d039d6949ed5ae"}
Oct 30 17:12:20.645949 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.645935    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"82a70ed30c47c86a6a1aee910e1d6754d01daee206a71e6cdcfca8ede0569844"}
Oct 30 17:12:20.646046 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.645965    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"fddd2e77cf18f276a5c723dabec5d1337ea0ab50aa00f9777d873e0da7e9cf37"}
Oct 30 17:12:20.649047 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.649025    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" event={"ID":"34ccfea3-7a41-4925-9669-c4a25efabc4a","Type":"ContainerStarted","Data":"ede3d114a86025c4e536d4625fc994dcdb316699167dbfa1bc098b9e6ec9141a"}
Oct 30 17:12:20.693190 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.693165    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:20.699069 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:20.699034    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5xbwb" podStartSLOduration=5.972319911 podStartE2EDuration="16.699022625s" totalImagesPullingTime="10.726702714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:20.698667749 +0000 UTC m=+17.840785724" watchObservedRunningTime="2026-10-30 17:12:20.699022625 +0000 UTC m=+17.841140605"
Oct 30 17:12:21.257474 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:21.257303    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xjh5h_7a3c7100-3ff5-4410-abd7-03e68c27baa3/node-ca/0.log"
Oct 30 17:12:21.654250 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:21.654211    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"8db5815142845854bf30d213f2a35eab04d96056e4e4ff667e44d6a0a2c1f7d5"}
Oct 30 17:12:21.654396 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:21.654295    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"00cdef5387e730b7cfb7ca0c89cdcaefee60e963026fa0c518f8561e38377935"}
Oct 30 17:12:21.709592 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:21.709555    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:21.819825 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:21.819793    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:21.925367 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:21.925276    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:22.095581 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.095542    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:22.299722 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.299615    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:22.388633 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.388604    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:22.388633 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.388632    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:22.388851 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:22.388729    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:22.388851 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:22.388791    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:22.388851 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.388842    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:22.388945 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:22.388891    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:22.417539 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.417507    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:22.531281 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.531240    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:22.694126 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.694096    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:22.954836 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:22.954767    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:23.379171 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:23.379135    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:23.491576 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:23.491544    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:23.661578 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:23.661495    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"571a23d8ebe04239bc5eb0fc24871bbe6e8f9faed424208ef99fe95ed6c07aa8"}
Oct 30 17:12:23.895631 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:23.895595    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.089564 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.089476    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.203028 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.202992    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.319052 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.319019    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.388976 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.388939    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:24.389337 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.389005    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:24.389337 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.389102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:24.389337 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:24.389105    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:24.389337 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:24.389166    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:24.389337 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:24.389254    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:24.438455 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.438415    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.544625 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.544594    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.658001 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.657912    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:24.765321 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:24.765294    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:25.669115 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:25.669078    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" event={"ID":"84532c65-2d04-45a5-b0cb-9841d8e20d87","Type":"ContainerStarted","Data":"fb328136d08e995ab0530929c371019bce09113e35d742380b02cb9eee678025"}
Oct 30 17:12:25.669911 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:25.669353    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:25.669911 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:25.669431    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:25.684703 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:25.684685    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:25.689502 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:25.689478    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:25.763166 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:25.763112    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd" podStartSLOduration=6.106018447 podStartE2EDuration="21.76309908s" totalImagesPullingTime="15.657080633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:25.762454047 +0000 UTC m=+22.904572027" watchObservedRunningTime="2026-10-30 17:12:25.76309908 +0000 UTC m=+22.905217091"
Oct 30 17:12:26.104424 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.104391    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:26.216921 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.216889    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:26.331044 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.331007    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:26.388911 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.388831    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:26.388911 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.388861    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:26.389104 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:26.388950    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:26.389104 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.388968    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:26.389104 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:26.389038    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:26.389240 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:26.389109    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:26.492451 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.492410    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:26.671320 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.671240    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:26.676562 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.676531    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:26.686569 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.686550    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:26.789411 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:26.789373    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:27.892774 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:27.892739    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:28.082370 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.082328    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:28.194450 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.194379    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:28.300907 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.300875    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:28.388829 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.388798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:28.388969 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.388799    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:28.388969 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:28.388901    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:28.388969 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.388935    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:28.389059 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:28.389009    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:28.389096 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:28.389059    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:28.492172 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.492094    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:28.695915 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:28.695881    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:29.156121 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:29.156086    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:29.890676 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:29.890629    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:30.088667 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.088615    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:30.291043 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.290959    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:30.388803 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.388772    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:30.388803 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.388811    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:30.388989 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.388811    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:30.388989 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:30.388899    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:30.389055 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:30.389019    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:30.389112 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:30.389094    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:30.489585 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.489557    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:30.598934 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.598905    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:30.709112 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:30.709077    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:32.388920 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:32.388891    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:32.389310 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:32.388982    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:32.389310 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:32.388990    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:32.389310 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:32.388990    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:32.389310 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:32.389071    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:32.389310 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:32.389126    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:34.388552 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:34.388513    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:34.388552 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:34.388555    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:34.388992 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:34.388560    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:34.388992 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:34.388636    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:34.388992 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:34.388738    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:34.388992 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:34.388797    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:36.083533 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.083491    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:36.084098 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.083546    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:36.084098 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.083664    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:36.084098 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.083684    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:36.084098 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.083747    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret podName:f3dcc262-c93f-43f7-ba5f-601ac5ab06b3 nodeName:}" failed. No retries permitted until 2026-10-30 17:13:08.08372808 +0000 UTC m=+65.225846038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret") pod "global-pull-secret-syncer-v8rgj" (UID: "f3dcc262-c93f-43f7-ba5f-601ac5ab06b3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 17:12:36.084098 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.083767    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs podName:a5a1e416-7f8a-4940-85c2-19f0fd39c5f4 nodeName:}" failed. No retries permitted until 2026-10-30 17:13:08.083756192 +0000 UTC m=+65.225874149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs") pod "network-metrics-daemon-dqkvr" (UID: "a5a1e416-7f8a-4940-85c2-19f0fd39c5f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 17:12:36.184088 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.184050    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:36.184241 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.184211    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 17:12:36.184241 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.184230    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 17:12:36.184241 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.184241    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drb5c for pod openshift-network-diagnostics/network-check-target-z2hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:36.184361 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.184293    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c podName:cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a nodeName:}" failed. No retries permitted until 2026-10-30 17:13:08.184279765 +0000 UTC m=+65.326397735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-drb5c" (UniqueName: "kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c") pod "network-check-target-z2hmf" (UID: "cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 17:12:36.388873 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.388840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:36.389057 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.388840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:36.389057 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.388967    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:36.389057 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.388840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:36.389057 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.389016    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:36.389254 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:36.389109    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:36.800399 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.800313    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:36.914948 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:36.914902    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:37.034069 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:37.034036    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:38.028896 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:38.028855    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8rgj"]
Oct 30 17:12:38.029415 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:38.029040    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:38.029415 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:38.029163    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:38.032992 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:38.032959    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2hmf"]
Oct 30 17:12:38.033144 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:38.033099    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:38.033233 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:38.033209    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:38.033819 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:38.033795    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqkvr"]
Oct 30 17:12:38.033923 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:38.033911    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:38.034005 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:38.033990    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:39.389218 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:39.389185    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:39.389218 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:39.389215    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:39.389606 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:39.389190    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:39.389606 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:39.389297    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:39.389606 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:39.389371    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:39.389606 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:39.389442    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:39.475105 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:39.475072    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:39.593863 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:39.593839    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:40.731039 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:40.731018    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zz6f9_991f66af-6bcb-4bd8-921f-58f2fb879067/dns-node-resolver/0.log"
Oct 30 17:12:41.388462 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:41.388431    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:41.388674 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:41.388442    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:41.388674 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:41.388550    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:41.388674 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:41.388549    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:41.388799 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:41.388671    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:41.388839 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:41.388789    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:41.931045 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:41.931017    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xjh5h_7a3c7100-3ff5-4410-abd7-03e68c27baa3/node-ca/0.log"
Oct 30 17:12:42.343590 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:42.343548    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:43.389985 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.389954    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:43.390385 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:43.390036    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:43.390385 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.390142    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:43.390385 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:43.390239    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:43.390385 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.390247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:43.390385 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:43.390301    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:43.560519 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.560488    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:43.702804 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.702739    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:43.816787 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.816752    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:43.927883 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:43.927855    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:44.191231 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:44.191205    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:44.371708 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:44.371637    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:44.729171 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:44.729139    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:44.837821 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:44.837785    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:45.388908 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:45.388877    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:45.389068 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:45.388919    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:45.389068 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:45.388882    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:45.389068 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:45.389013    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:45.389182 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:45.389068    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:45.389182 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:45.389127    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:45.838088 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:45.838010    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:45.950069 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:45.950019    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:47.388894 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:47.388864    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:47.388894 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:47.388880    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:47.389289 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:47.388999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:47.389289 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:47.389002    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:47.389289 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:47.389089    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:47.389289 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:47.389169    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:48.552663 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:48.552478    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:48.670035 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:48.670004    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:49.391276 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:49.391242    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:49.391453 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:49.391295    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:49.391453 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:49.391308    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:49.391453 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:49.391395    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:49.391453 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:49.391432    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:49.391705 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:49.391510    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:50.268997 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:50.268959    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:50.383972 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:50.383943    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:50.498610 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:50.498569    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:51.388534 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.388500    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:51.389021 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.388501    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:51.389021 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.388664    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:51.389021 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:51.388662    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqkvr" podUID="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4"
Oct 30 17:12:51.389021 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:51.388770    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2hmf" podUID="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a"
Oct 30 17:12:51.389021 ip-10-0-135-170 kubenswrapper[2570]: E1030 17:12:51.388861    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8rgj" podUID="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3"
Oct 30 17:12:51.575247 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.575221    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeReady"
Oct 30 17:12:51.575378 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.575366    2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 17:12:51.642042 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.641970    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rcllv"]
Oct 30 17:12:51.643857 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.643843    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.649571 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.649545    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p274z"]
Oct 30 17:12:51.651575 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.651186    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.662031 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.662006    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 17:12:51.662122 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.662076    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rz5ks\""
Oct 30 17:12:51.662122 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.662089    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 17:12:51.671910 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.671887    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pwswg"]
Oct 30 17:12:51.673481 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.673457    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.674963 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.674940    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcllv"]
Oct 30 17:12:51.676436 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.676415    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p274z"]
Oct 30 17:12:51.696259 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.696239    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pwswg"]
Oct 30 17:12:51.710962 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.710918    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:51.761619 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.761583    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 17:12:51.761619 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.761590    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wklnr\""
Oct 30 17:12:51.761619 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.761626    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 17:12:51.761864 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.761696    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zkp55\""
Oct 30 17:12:51.762525 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.762501    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 17:12:51.762525 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.762517    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 17:12:51.762525 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.762527    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 17:12:51.762714 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.762536    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 17:12:51.762714 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.762558    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 17:12:51.802134 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa7322d-ff57-4460-be07-755ae180ffe4-crio-socket\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.802134 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802137    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4hz\" (UniqueName: \"kubernetes.io/projected/a6b374b9-fd59-4579-8ebd-fe975116467c-kube-api-access-9m4hz\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.802331 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802171    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa7322d-ff57-4460-be07-755ae180ffe4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.802331 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802233    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a516e5-f016-4d54-be70-b1979fe0c605-config-volume\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.802331 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a516e5-f016-4d54-be70-b1979fe0c605-tmp-dir\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.802421 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802374    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa7322d-ff57-4460-be07-755ae180ffe4-data-volume\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.802421 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zh6\" (UniqueName: \"kubernetes.io/projected/caa7322d-ff57-4460-be07-755ae180ffe4-kube-api-access-g4zh6\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.802421 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6b374b9-fd59-4579-8ebd-fe975116467c-cert\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.802504 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6a516e5-f016-4d54-be70-b1979fe0c605-metrics-tls\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.802540 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802497    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa7322d-ff57-4460-be07-755ae180ffe4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.802540 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.802526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl9k9\" (UniqueName: \"kubernetes.io/projected/a6a516e5-f016-4d54-be70-b1979fe0c605-kube-api-access-tl9k9\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.827470 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.827438    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:51.903529 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903441    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa7322d-ff57-4460-be07-755ae180ffe4-crio-socket\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.903529 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4hz\" (UniqueName: \"kubernetes.io/projected/a6b374b9-fd59-4579-8ebd-fe975116467c-kube-api-access-9m4hz\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.903529 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903513    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa7322d-ff57-4460-be07-755ae180ffe4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.903529 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903535    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a516e5-f016-4d54-be70-b1979fe0c605-config-volume\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903580    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a516e5-f016-4d54-be70-b1979fe0c605-tmp-dir\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903616    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa7322d-ff57-4460-be07-755ae180ffe4-data-volume\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903666    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zh6\" (UniqueName: \"kubernetes.io/projected/caa7322d-ff57-4460-be07-755ae180ffe4-kube-api-access-g4zh6\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903693    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa7322d-ff57-4460-be07-755ae180ffe4-crio-socket\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903706    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6b374b9-fd59-4579-8ebd-fe975116467c-cert\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903741    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6a516e5-f016-4d54-be70-b1979fe0c605-metrics-tls\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903778    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa7322d-ff57-4460-be07-755ae180ffe4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.903924 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.903807    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9k9\" (UniqueName: \"kubernetes.io/projected/a6a516e5-f016-4d54-be70-b1979fe0c605-kube-api-access-tl9k9\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.904261 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.904245    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a516e5-f016-4d54-be70-b1979fe0c605-tmp-dir\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.904324 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.904300    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a516e5-f016-4d54-be70-b1979fe0c605-config-volume\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.904359 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.904302    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa7322d-ff57-4460-be07-755ae180ffe4-data-volume\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.904468 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.904441    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa7322d-ff57-4460-be07-755ae180ffe4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.907658 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.907619    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6a516e5-f016-4d54-be70-b1979fe0c605-metrics-tls\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.907747 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.907623    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa7322d-ff57-4460-be07-755ae180ffe4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.907798 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.907775    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6b374b9-fd59-4579-8ebd-fe975116467c-cert\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.928219 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.928195    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9k9\" (UniqueName: \"kubernetes.io/projected/a6a516e5-f016-4d54-be70-b1979fe0c605-kube-api-access-tl9k9\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.928329 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.928313    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4hz\" (UniqueName: \"kubernetes.io/projected/a6b374b9-fd59-4579-8ebd-fe975116467c-kube-api-access-9m4hz\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:51.928493 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.928472    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zh6\" (UniqueName: \"kubernetes.io/projected/caa7322d-ff57-4460-be07-755ae180ffe4-kube-api-access-g4zh6\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.955450 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.955425    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:51.962805 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.962783    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:12:51.984477 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:51.984450    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:12:52.232744 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.232625    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:52.265108 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.265087    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcllv"]
Oct 30 17:12:52.269503 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:52.268219    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a516e5_f016_4d54_be70_b1979fe0c605.slice/crio-e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4 WatchSource:0}: Error finding container e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4: Status 404 returned error can't find the container with id e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4
Oct 30 17:12:52.278357 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.278336    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p274z"]
Oct 30 17:12:52.280769 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:52.280747    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa7322d_ff57_4460_be07_755ae180ffe4.slice/crio-3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9 WatchSource:0}: Error finding container 3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9: Status 404 returned error can't find the container with id 3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9
Oct 30 17:12:52.299527 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.299507    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pwswg"]
Oct 30 17:12:52.307531 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:52.307509    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b374b9_fd59_4579_8ebd_fe975116467c.slice/crio-029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a WatchSource:0}: Error finding container 029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a: Status 404 returned error can't find the container with id 029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a
Oct 30 17:12:52.358541 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.358500    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:52.727417 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.727384    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pwswg" event={"ID":"a6b374b9-fd59-4579-8ebd-fe975116467c","Type":"ContainerStarted","Data":"029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a"}
Oct 30 17:12:52.729299 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.729266    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p274z" event={"ID":"caa7322d-ff57-4460-be07-755ae180ffe4","Type":"ContainerStarted","Data":"aa22c1f46dad5c8d636ed68811028095a7192304141d653960b5c3ff934d27c7"}
Oct 30 17:12:52.729464 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.729321    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p274z" event={"ID":"caa7322d-ff57-4460-be07-755ae180ffe4","Type":"ContainerStarted","Data":"3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9"}
Oct 30 17:12:52.730728 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:52.730699    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcllv" event={"ID":"a6a516e5-f016-4d54-be70-b1979fe0c605","Type":"ContainerStarted","Data":"e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4"}
Oct 30 17:12:53.391300 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.389948    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:12:53.391300 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.390715    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:12:53.391510 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.391405    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:12:53.449412 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.449350    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 17:12:53.449412 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.449360    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 17:12:53.462725 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.462700    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 17:12:53.462725 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.462722    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\""
Oct 30 17:12:53.462917 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.462876    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\""
Oct 30 17:12:53.462917 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.462896    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\""
Oct 30 17:12:53.463236 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.463217    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 17:12:53.736133 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:53.736048    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p274z" event={"ID":"caa7322d-ff57-4460-be07-755ae180ffe4","Type":"ContainerStarted","Data":"3b021d4eab0473a7551ac4e7bf8eee84395e61d0f59755308fced9d59b77a352"}
Oct 30 17:12:54.305001 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.304099    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:54.429547 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.429512    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:54.741191 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.741103    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p274z" event={"ID":"caa7322d-ff57-4460-be07-755ae180ffe4","Type":"ContainerStarted","Data":"662e459daa57b5ee52f965a6c5195f09d9570386788d7df198e4f5e8e1270377"}
Oct 30 17:12:54.743261 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.743233    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcllv" event={"ID":"a6a516e5-f016-4d54-be70-b1979fe0c605","Type":"ContainerStarted","Data":"19608b6120eee8a18b9f0b32f9e12c6a904d6c932aeb6f7dbd8c92201e3f1a73"}
Oct 30 17:12:54.743394 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.743286    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcllv" event={"ID":"a6a516e5-f016-4d54-be70-b1979fe0c605","Type":"ContainerStarted","Data":"c5248e15325ad87f5f2ae53d628e790754fe24eaa655aaa680f6c575e4bfb673"}
Oct 30 17:12:54.743394 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.743344    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rcllv"
Oct 30 17:12:54.847482 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.847382    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rcllv" podStartSLOduration=2.045719201 podStartE2EDuration="3.847028314s" totalImagesPullingTime="1.801309113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:54.846406336 +0000 UTC m=+51.988524315" watchObservedRunningTime="2026-10-30 17:12:54.847028314 +0000 UTC m=+51.989146294"
Oct 30 17:12:54.847715 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.847615    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p274z" podStartSLOduration=1.6740206180000001 podStartE2EDuration="3.847600461s" totalImagesPullingTime="2.173579843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:54.800097844 +0000 UTC m=+51.942215823" watchObservedRunningTime="2026-10-30 17:12:54.847600461 +0000 UTC m=+51.989718440"
Oct 30 17:12:54.920615 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:54.920582    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:55.746863 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:55.746817    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pwswg" event={"ID":"a6b374b9-fd59-4579-8ebd-fe975116467c","Type":"ContainerStarted","Data":"9d457dda69714becc9bf02e90d1e623fd02167bc03d3693c1cc18cc1c5070053"}
Oct 30 17:12:55.795614 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:55.795563    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pwswg" podStartSLOduration=2.158215725 podStartE2EDuration="4.795550389s" totalImagesPullingTime="2.637334664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:55.794999045 +0000 UTC m=+52.937117025" watchObservedRunningTime="2026-10-30 17:12:55.795550389 +0000 UTC m=+52.937668362"
Oct 30 17:12:56.690739 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.690695    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:56.896033 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.895993    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:56.962116 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.962050    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"]
Oct 30 17:12:56.965176 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.965162    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:56.998141 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.998120    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 17:12:56.998267 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.998173    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 17:12:56.998267 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.998218    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 17:12:56.998342 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.998332    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 17:12:56.998595 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.998578    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 17:12:56.998655 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:56.998624    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x2mz9\""
Oct 30 17:12:57.001230 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.001212    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"]
Oct 30 17:12:57.002164 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.002144    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mbfgv"]
Oct 30 17:12:57.005088 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.005074    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.012400 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.012384    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"]
Oct 30 17:12:57.015832 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.015814    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.023327 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.023306    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 17:12:57.023397 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.023342    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-726zt\""
Oct 30 17:12:57.023397 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.023385    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 17:12:57.030567 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.030534    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"]
Oct 30 17:12:57.031745 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.031718    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:57.046281 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.046259    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.046363 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.046297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.046363 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.046326    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/376a21c0-10de-43a4-a04c-c3519a868a9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.046454 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.046436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8wh\" (UniqueName: \"kubernetes.io/projected/376a21c0-10de-43a4-a04c-c3519a868a9b-kube-api-access-rb8wh\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.057759 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.057741    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 17:12:57.058187 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.058144    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 17:12:57.061736 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.061720    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 17:12:57.061850 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.061839    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 17:12:57.061902 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.061886    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cr2cr\""
Oct 30 17:12:57.138933 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.138904    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:57.146721 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146705    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.146775 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.146775 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/376a21c0-10de-43a4-a04c-c3519a868a9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.146775 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-wtmp\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.146863 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-metrics-client-ca\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.146897 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.146986 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146975    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8wh\" (UniqueName: \"kubernetes.io/projected/376a21c0-10de-43a4-a04c-c3519a868a9b-kube-api-access-rb8wh\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.147025 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.146996    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.147025 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.147093 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.147093 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147067    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-tls\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.147153 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-textfile\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.147205 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147169    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.147296 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147270    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-root\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.147767 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/908859a0-1e77-4a62-a253-3486242475d8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.147849 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-sys\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.147849 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147814    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlt82\" (UniqueName: \"kubernetes.io/projected/322b7165-7398-4ed9-99a9-cef5e8ee3c86-kube-api-access-wlt82\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.147955 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhq7\" (UniqueName: \"kubernetes.io/projected/908859a0-1e77-4a62-a253-3486242475d8-kube-api-access-6vhq7\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.147955 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147923    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/376a21c0-10de-43a4-a04c-c3519a868a9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.147955 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.147947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.149712 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.149694    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.149930 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.149916    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.171391 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.171367    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8wh\" (UniqueName: \"kubernetes.io/projected/376a21c0-10de-43a4-a04c-c3519a868a9b-kube-api-access-rb8wh\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.248779    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-tls\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.248886    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-textfile\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.248932    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-root\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.248972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/908859a0-1e77-4a62-a253-3486242475d8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-sys\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249064    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlt82\" (UniqueName: \"kubernetes.io/projected/322b7165-7398-4ed9-99a9-cef5e8ee3c86-kube-api-access-wlt82\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhq7\" (UniqueName: \"kubernetes.io/projected/908859a0-1e77-4a62-a253-3486242475d8-kube-api-access-6vhq7\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249167    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249277    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-wtmp\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-metrics-client-ca\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249358    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.249391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.252280 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.250076    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-wtmp\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.250254    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-root\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.250410    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-sys\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.250550    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-textfile\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.250908    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/908859a0-1e77-4a62-a253-3486242475d8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.251416    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.252288    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.253092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.252841    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-metrics-client-ca\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.253405 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.253091    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.253405 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.253354    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.254291 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.254262    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-tls\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.255707 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.255672    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:57.255988 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.255966    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.256223 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.256202    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.273405 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.273387    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhq7\" (UniqueName: \"kubernetes.io/projected/908859a0-1e77-4a62-a253-3486242475d8-kube-api-access-6vhq7\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.274942 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.274924    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlt82\" (UniqueName: \"kubernetes.io/projected/322b7165-7398-4ed9-99a9-cef5e8ee3c86-kube-api-access-wlt82\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.279568 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.279555    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:12:57.320134 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.320108    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:12:57.330709 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.330685    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:12:57.332979 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:57.332952    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322b7165_7398_4ed9_99a9_cef5e8ee3c86.slice/crio-fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249 WatchSource:0}: Error finding container fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249: Status 404 returned error can't find the container with id fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249
Oct 30 17:12:57.394150 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.394009    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:57.586667 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.586630    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"]
Oct 30 17:12:57.588908 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:57.588880    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376a21c0_10de_43a4_a04c_c3519a868a9b.slice/crio-8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0 WatchSource:0}: Error finding container 8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0: Status 404 returned error can't find the container with id 8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0
Oct 30 17:12:57.637242 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.637212    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"]
Oct 30 17:12:57.640125 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:12:57.640104    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908859a0_1e77_4a62_a253_3486242475d8.slice/crio-d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff WatchSource:0}: Error finding container d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff: Status 404 returned error can't find the container with id d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff
Oct 30 17:12:57.754102 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.754067    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l" event={"ID":"376a21c0-10de-43a4-a04c-c3519a868a9b","Type":"ContainerStarted","Data":"d9a2f04118498aacccad532c1b3d75f4d6e9c9ec4190883db3c3f6d62ae7c067"}
Oct 30 17:12:57.754286 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.754125    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l" event={"ID":"376a21c0-10de-43a4-a04c-c3519a868a9b","Type":"ContainerStarted","Data":"1954c3f34e4befa1870cfd5c1d6a6335676156d48c84c84ea9a8f4c83a2799df"}
Oct 30 17:12:57.754286 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.754153    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l" event={"ID":"376a21c0-10de-43a4-a04c-c3519a868a9b","Type":"ContainerStarted","Data":"8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0"}
Oct 30 17:12:57.755249 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.755226    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbfgv" event={"ID":"322b7165-7398-4ed9-99a9-cef5e8ee3c86","Type":"ContainerStarted","Data":"fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249"}
Oct 30 17:12:57.756331 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:57.756313    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg" event={"ID":"908859a0-1e77-4a62-a253-3486242475d8","Type":"ContainerStarted","Data":"d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff"}
Oct 30 17:12:58.221406 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.221347    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:58.361240 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.361212    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:58.480628 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.480529    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:58.688432 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.688086    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:12:58.764824 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.764786    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l" event={"ID":"376a21c0-10de-43a4-a04c-c3519a868a9b","Type":"ContainerStarted","Data":"9023d86feb14a027477fd305d48cf3496d2bb031d3ef00c3673f38cfef04ebac"}
Oct 30 17:12:58.766702 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.766670    2570 generic.go:356] "Generic (PLEG): container finished" podID="322b7165-7398-4ed9-99a9-cef5e8ee3c86" containerID="0114a4c56a9ee68353246b697f60d4a82f8224cdc7350ea64a88a24eed10602c" exitCode=0
Oct 30 17:12:58.766864 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.766766    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbfgv" event={"ID":"322b7165-7398-4ed9-99a9-cef5e8ee3c86","Type":"ContainerDied","Data":"0114a4c56a9ee68353246b697f60d4a82f8224cdc7350ea64a88a24eed10602c"}
Oct 30 17:12:58.810565 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:58.810510    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l" podStartSLOduration=1.816237503 podStartE2EDuration="2.810495415s" totalImagesPullingTime="994.257912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:58.809405835 +0000 UTC m=+55.951523815" watchObservedRunningTime="2026-10-30 17:12:58.810495415 +0000 UTC m=+55.952613393"
Oct 30 17:12:59.071212 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.071186    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:59.183145 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.183073    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:59.495559 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.495518    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:59.605144 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.605110    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:59.719018 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.718981    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:12:59.771581 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.771502    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg" event={"ID":"908859a0-1e77-4a62-a253-3486242475d8","Type":"ContainerStarted","Data":"c6b38c50b004d132f715e495e8c941447517f9d1ef6db1d8cdb9db51264719a7"}
Oct 30 17:12:59.771581 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.771557    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg" event={"ID":"908859a0-1e77-4a62-a253-3486242475d8","Type":"ContainerStarted","Data":"761b7f4144a63a86499207dad0467cdd66847733dab97f413ec4124af8a49009"}
Oct 30 17:12:59.771787 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.771583    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg" event={"ID":"908859a0-1e77-4a62-a253-3486242475d8","Type":"ContainerStarted","Data":"76d44ee40bb362e3871cfc735aba348aac63245e575778b2c76d5ede91c26f17"}
Oct 30 17:12:59.773591 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.773570    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbfgv" event={"ID":"322b7165-7398-4ed9-99a9-cef5e8ee3c86","Type":"ContainerStarted","Data":"abe682654cd95baeaca688c9132aa740493df08aaa70c0a0fdaf2ff0e983fbf2"}
Oct 30 17:12:59.773680 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.773615    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbfgv" event={"ID":"322b7165-7398-4ed9-99a9-cef5e8ee3c86","Type":"ContainerStarted","Data":"7e1052170d4db72a09d94aa538bafb80451575feb27cad2ba2cf05a6cecf1ecb"}
Oct 30 17:12:59.817567 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.817521    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg" podStartSLOduration=2.516345211 podStartE2EDuration="3.817508407s" totalImagesPullingTime="1.301163196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:59.816866698 +0000 UTC m=+56.958984678" watchObservedRunningTime="2026-10-30 17:12:59.817508407 +0000 UTC m=+56.959626386"
Oct 30 17:12:59.862697 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:12:59.862631    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mbfgv" podStartSLOduration=2.7974511140000002 podStartE2EDuration="3.862614531s" totalImagesPullingTime="982.168167ms" totalInitContainerRuntime="82.99525ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:59.861547026 +0000 UTC m=+57.003665027" watchObservedRunningTime="2026-10-30 17:12:59.862614531 +0000 UTC m=+57.004732512"
Oct 30 17:13:00.263790 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:00.263753    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:00.380282 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:00.380238    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:00.509092 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:00.509065    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:00.618433 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:00.618407    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:01.806176 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.806142    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:01.831846 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.831823    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"]
Oct 30 17:13:01.834967 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.834950    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:01.864561 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.864531    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 17:13:01.864561 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.864552    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w2nmx\""
Oct 30 17:13:01.865440 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.865423    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"]
Oct 30 17:13:01.921926 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.921890    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:01.989022 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.988992    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccb7x\" (UniqueName: \"kubernetes.io/projected/df1c926e-5a6b-4c44-881c-5dc47879a022-kube-api-access-ccb7x\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:01.989153 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:01.989028    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c926e-5a6b-4c44-881c-5dc47879a022-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:02.030168 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.030131    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:02.090390 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.090360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccb7x\" (UniqueName: \"kubernetes.io/projected/df1c926e-5a6b-4c44-881c-5dc47879a022-kube-api-access-ccb7x\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:02.090543 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.090396    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c926e-5a6b-4c44-881c-5dc47879a022-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:02.093068 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.093032    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c926e-5a6b-4c44-881c-5dc47879a022-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:02.114438 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.114410    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccb7x\" (UniqueName: \"kubernetes.io/projected/df1c926e-5a6b-4c44-881c-5dc47879a022-kube-api-access-ccb7x\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:02.145513 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.145476    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:02.279224 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.279190    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"]
Oct 30 17:13:02.283529 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.283504    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.299811 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.299781    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 17:13:02.299811 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.299781    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 17:13:02.299998 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.299808    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 17:13:02.299998 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.299822    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 17:13:02.312563 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.312522    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zkwh9\""
Oct 30 17:13:02.312757 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.312737    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 17:13:02.326487 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.326459    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 17:13:02.327935 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.327918    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"]
Oct 30 17:13:02.394060 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.393966    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394060 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxqx\" (UniqueName: \"kubernetes.io/projected/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-kube-api-access-srxqx\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394279 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394116    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394279 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-metrics-client-ca\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394279 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394279 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394260    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-federate-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394416 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394278    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-serving-certs-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.394416 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.394353    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.445200 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.445171    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"]
Oct 30 17:13:02.447711 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:13:02.447686    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1c926e_5a6b_4c44_881c_5dc47879a022.slice/crio-d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba WatchSource:0}: Error finding container d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba: Status 404 returned error can't find the container with id d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba
Oct 30 17:13:02.495275 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495243    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-metrics-client-ca\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495275 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495277    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495469 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-federate-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495469 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495332    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-serving-certs-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495469 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495469 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495469 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495449    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxqx\" (UniqueName: \"kubernetes.io/projected/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-kube-api-access-srxqx\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.495717 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.495480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.496332 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.496112    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-metrics-client-ca\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.496332 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.496193    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-serving-certs-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.496332 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.496250    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.497996 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.497976    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.498117 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.498100    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.498535 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.498505    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.498578 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.498529    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-federate-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.521635 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.521605    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxqx\" (UniqueName: \"kubernetes.io/projected/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-kube-api-access-srxqx\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.598604 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.598565    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:02.783914 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.783879    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v" event={"ID":"df1c926e-5a6b-4c44-881c-5dc47879a022","Type":"ContainerStarted","Data":"d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba"}
Oct 30 17:13:02.937827 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:02.937751    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"]
Oct 30 17:13:02.940617 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:13:02.940590    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c86d1f6_23f6_4d70_bcac_400d7c08bc09.slice/crio-dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd WatchSource:0}: Error finding container dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd: Status 404 returned error can't find the container with id dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd
Oct 30 17:13:03.646444 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:03.646404    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:03.789047 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:03.789008    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t" event={"ID":"4c86d1f6-23f6-4d70-bcac-400d7c08bc09","Type":"ContainerStarted","Data":"dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd"}
Oct 30 17:13:03.848029 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:03.847990    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:04.027352 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.027271    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 17:13:04.029869 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.029838    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:04.031339 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.031320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.048023 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.047997    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 17:13:04.048223 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.048207    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 17:13:04.048548 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.048349    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 17:13:04.048548 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.048379    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 17:13:04.048548 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.048409    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 17:13:04.061247 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061227    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 17:13:04.061247 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061227    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 17:13:04.061387 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061272    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jw6df\""
Oct 30 17:13:04.061668 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061620    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 17:13:04.061668 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061656    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 17:13:04.061826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061672    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 17:13:04.061826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061696    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 17:13:04.061826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061664    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 17:13:04.061826 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061711    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 17:13:04.062003 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.061958    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3n2877180h8a6\""
Oct 30 17:13:04.089085 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.089054    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 17:13:04.089197 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.089180    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 17:13:04.090565 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.090543    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 17:13:04.193228 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.193177    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:04.209130 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209262 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209150    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209262 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209172    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209351 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209385 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209369    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209439 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209425    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209487 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209473    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209523 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209508    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209556 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209540    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209584 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209627 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209607    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209715 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209699    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209752 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209794 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209772    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209879 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209861    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209911 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209893    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209961 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.209999 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.209975    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxr4\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-kube-api-access-8gxr4\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.210028 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.210009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.210063 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.210033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311260 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311260 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311363    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxr4\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-kube-api-access-8gxr4\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311410    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311483 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311471    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311491    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311581    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311610    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311696    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311758    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.311888 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.311793    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.312315 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.312106    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.312829 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.312798    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.314953 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.314344    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.314953 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.314564    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.314953 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.314823    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.315164 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.314962    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.315773 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.315623    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.315882 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.315792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.316100 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.316052    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.316175 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.316103    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.318059 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.317429    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.318059 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.318009    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.318542 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.318515    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.318799 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.318780    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.318870 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.318777    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.319205 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.319159    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.320222 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.320198    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.320350 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.320333    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.320818 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.320796    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.336505 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.336487    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxr4\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-kube-api-access-8gxr4\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.339107 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.339085    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:04.359260 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.359243    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:04.708292 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.708260    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 17:13:04.724001 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:13:04.723972    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dca0b75_8ca5_4e65_8c40_75ca747fffb3.slice/crio-ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc WatchSource:0}: Error finding container ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc: Status 404 returned error can't find the container with id ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc
Oct 30 17:13:04.749569 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.749539    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:04.793854 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.793818    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v" event={"ID":"df1c926e-5a6b-4c44-881c-5dc47879a022","Type":"ContainerStarted","Data":"f209ffd88ec8130cce1c9189b7e692a93303388c64a0e9c407f29d13144e111f"}
Oct 30 17:13:04.794006 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.793987    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:04.795706 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.795665    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t" event={"ID":"4c86d1f6-23f6-4d70-bcac-400d7c08bc09","Type":"ContainerStarted","Data":"c8b9b1fa6166a5c9874369dde1b67816e190d6a93ebec73bf6c23c55fbfd1d2c"}
Oct 30 17:13:04.797198 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.797180    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc"}
Oct 30 17:13:04.799786 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.799769    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:04.848813 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:04.848768    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v" podStartSLOduration=2.506030292 podStartE2EDuration="3.848754722s" totalImagesPullingTime="1.34272443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:04.847811124 +0000 UTC m=+61.989929114" watchObservedRunningTime="2026-10-30 17:13:04.848754722 +0000 UTC m=+61.990872700"
Oct 30 17:13:05.013380 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:05.013307    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:05.120800 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:05.120752    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:05.802320 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:05.802286    2570 generic.go:356] "Generic (PLEG): container finished" podID="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" containerID="875df2990e863457c688117c7ab8168564010d44d0b5990e32141b46af0fa6df" exitCode=0
Oct 30 17:13:05.802459 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:05.802372    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerDied","Data":"875df2990e863457c688117c7ab8168564010d44d0b5990e32141b46af0fa6df"}
Oct 30 17:13:06.691453 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:06.691409    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:06.812870 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:06.811924    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:06.815297 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:06.815262    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t" event={"ID":"4c86d1f6-23f6-4d70-bcac-400d7c08bc09","Type":"ContainerStarted","Data":"6e462ad9ddb81b40e279c4f00c2a2906b7f1d379259b94995512f0bdcaeba716"}
Oct 30 17:13:06.815416 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:06.815360    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t" event={"ID":"4c86d1f6-23f6-4d70-bcac-400d7c08bc09","Type":"ContainerStarted","Data":"c936aea622e7c4c36075c3acfe86cbee9b91da48182046ee8fdc6aa0df085d41"}
Oct 30 17:13:06.879156 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:06.879091    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t" podStartSLOduration=2.140043896 podStartE2EDuration="4.879069879s" totalImagesPullingTime="2.739025983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:06.864412752 +0000 UTC m=+64.006530764" watchObservedRunningTime="2026-10-30 17:13:06.879069879 +0000 UTC m=+64.021187863"
Oct 30 17:13:07.325367 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:07.325324    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:07.497391 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:07.497356    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:08.147658 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.147609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:08.148094 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.147699    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:08.166408 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.166379    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 17:13:08.167541 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.167519    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 17:13:08.171517 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.171492    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:08.171696 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.171549    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:08.248756 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.248716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:08.266842 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.266816    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 17:13:08.285245 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.285210    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 17:13:08.293119 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.293091    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:08.424842 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.424773    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\""
Oct 30 17:13:08.429864 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.429834    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:08.429993 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.429888    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\""
Oct 30 17:13:08.434745 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.434727    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\""
Oct 30 17:13:08.435718 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.435697    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:08.440599 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.440580    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:08.759014 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.758939    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqkvr"]
Oct 30 17:13:08.762228 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:13:08.762201    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a1e416_7f8a_4940_85c2_19f0fd39c5f4.slice/crio-c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e WatchSource:0}: Error finding container c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e: Status 404 returned error can't find the container with id c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e
Oct 30 17:13:08.770292 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.770267    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8rgj"]
Oct 30 17:13:08.773964 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.773936    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2hmf"]
Oct 30 17:13:08.774209 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:13:08.774180    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dcc262_c93f_43f7_ba5f_601ac5ab06b3.slice/crio-4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4 WatchSource:0}: Error finding container 4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4: Status 404 returned error can't find the container with id 4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4
Oct 30 17:13:08.775842 ip-10-0-135-170 kubenswrapper[2570]: W1030 17:13:08.775681    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad1bce8_8076_4ed0_a90d_73e5fd9b4b4a.slice/crio-fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4 WatchSource:0}: Error finding container fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4: Status 404 returned error can't find the container with id fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4
Oct 30 17:13:08.822433 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.822403    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8rgj" event={"ID":"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3","Type":"ContainerStarted","Data":"4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4"}
Oct 30 17:13:08.823581 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.823562    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqkvr" event={"ID":"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4","Type":"ContainerStarted","Data":"c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e"}
Oct 30 17:13:08.824624 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:08.824607    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2hmf" event={"ID":"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a","Type":"ContainerStarted","Data":"fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4"}
Oct 30 17:13:09.297786 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:09.297750    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:09.415232 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:09.415174    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:09.739526 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:09.739485    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:09.832753 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:09.832717    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"6ca0b6fa8ddc4188a544491e590ed84175773307f1c624cdb95611205dec7a14"}
Oct 30 17:13:09.832934 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:09.832803    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"4cddedeb59ebba657ed726197d8448fe079d31f62f0fc7dc2c8201bd4da1d6fd"}
Oct 30 17:13:09.885880 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:09.885836    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:10.593784 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:10.593632    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:10.701246 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:10.701212    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:10.849628 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:10.849028    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqkvr" event={"ID":"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4","Type":"ContainerStarted","Data":"65509175dadb05ff5e4716ed50e1214ab0d4e0db8b45cab9f12e1ccc3e375992"}
Oct 30 17:13:10.849628 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:10.849138    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqkvr" event={"ID":"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4","Type":"ContainerStarted","Data":"a4616051176ca4fc7519b44ca6464f0e506b8115752b557d86153799c8174da2"}
Oct 30 17:13:10.895782 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:10.894495    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dqkvr" podStartSLOduration=66.677376091 podStartE2EDuration="1m7.89447449s" totalImagesPullingTime="1.217098399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:10.893116626 +0000 UTC m=+68.035234605" watchObservedRunningTime="2026-10-30 17:13:10.89447449 +0000 UTC m=+68.036592469"
Oct 30 17:13:11.868889 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:11.868706    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"63de95babdb1df3acce8d174959e1483854b3ced6f27435ee5b7a64aaf05bb10"}
Oct 30 17:13:11.868889 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:11.868786    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"68748fe043562c4c77ba656b8ced23a5e4deb3ca1157c0ae8ece7bdc6c84c686"}
Oct 30 17:13:11.868889 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:11.868826    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"71e127f30b01fab4783727fef83aeba943e47797214683605ff88f769c61761d"}
Oct 30 17:13:11.868889 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:11.868861    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dca0b75-8ca5-4e65-8c40-75ca747fffb3","Type":"ContainerStarted","Data":"503c59b76a4d1d42773d5e3145309f4ae65c4b945763aba8577d7f5ab9a5c530"}
Oct 30 17:13:11.930719 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:11.930655    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.372373467 podStartE2EDuration="8.930619139s" totalImagesPullingTime="6.541409263s" totalInitContainerRuntime="16.836409ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:11.928375308 +0000 UTC m=+69.070493288" watchObservedRunningTime="2026-10-30 17:13:11.930619139 +0000 UTC m=+69.072737120"
Oct 30 17:13:12.123143 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:12.123102    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:12.260580 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:12.260541    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:12.476456 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:12.476367    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:12.587621 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:12.587596    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:12.874727 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:12.874688    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2hmf" event={"ID":"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a","Type":"ContainerStarted","Data":"a300d6e428392f95985e2f666e2b33107a76c6eedaa397ca0f4759db0f025ccd"}
Oct 30 17:13:12.875159 ip-10-0-135-170 kubenswrapper[2570]: I1030 17:13:12.874845    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:14.177782 ip-10-0-135-170 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 17:13:14.190989 ip-10-0-135-170 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 17:13:14.191202 ip-10-0-135-170 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 17:13:14.191260 ip-10-0-135-170 systemd[1]: kubelet.service: Consumed 4.252s CPU time.
Oct 30 17:13:14.199299 ip-10-0-135-170 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 17:13:14.270534 ip-10-0-135-170 kubenswrapper[5534]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 17:13:14.270534 ip-10-0-135-170 kubenswrapper[5534]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 17:13:14.270534 ip-10-0-135-170 kubenswrapper[5534]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 17:13:14.270534 ip-10-0-135-170 kubenswrapper[5534]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 17:13:14.272985 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.272902    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.272985 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.272983    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.272985 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.272989    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.272994    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.272999    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273005    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273010    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273015    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273019    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273023    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273027    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273032    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273036    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273041    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273045    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273050    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273054    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273058    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273063    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273067    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273080    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273085    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.273208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273090    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273094    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273099    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273103    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273108    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273112    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273117    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273122    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273127    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273133    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273138    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273143    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273147    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273152    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273157    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273161    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273166    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273172    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273177    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273183    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.274136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273188    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273192    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273197    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273202    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273206    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273211    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273215    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273220    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273225    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273229    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273234    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273238    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273243    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273248    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273253    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273258    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273263    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273268    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273273    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.275059 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273278    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273283    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273287    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273292    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273298    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273303    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273307    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273316    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273322    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273328    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273333    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273338    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273343    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273348    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273353    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273358    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273362    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273367    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273372    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.275922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273377    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273381    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273386    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273393    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273399    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273406    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273411    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273416    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273421    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273428    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273434    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273438    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273443    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273448    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273453    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273457    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273462    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273466    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273472    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.276817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273477    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273482    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273487    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273491    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273496    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273501    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.273505    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274784    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274795    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274800    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274805    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274810    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274816    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274821    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274826    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274832    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274837    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274841    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274846    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274851    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.277699 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274856    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274862    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274867    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274871    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274876    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274881    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274886    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274890    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274894    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274899    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274904    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274908    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274922    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274927    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274935    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274941    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274947    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274952    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274957    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274962    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.278614 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274967    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274972    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274977    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274982    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274987    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274992    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.274996    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275001    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275006    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275010    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275015    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275019    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275024    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275029    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275034    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275038    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275043    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275048    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275053    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275057    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.279564 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275061    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275066    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275070    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275074    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275079    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275092    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275098    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275103    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275107    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275112    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275117    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275121    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275126    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275131    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275135    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275140    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275145    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275151    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275156    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275161    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.280292 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275166    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275170    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275174    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275179    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275193    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275198    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275202    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275207    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275211    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275216    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275220    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275225    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275230    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275234    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275239    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275244    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275248    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275253    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275258    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.280828 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275263    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275268    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275277    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275283    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275289    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275294    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275300    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275304    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275309    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275314    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275319    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275324    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275328    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.275333    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276252    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276261    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276267    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276272    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276277    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.281351 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276282    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276287    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276291    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276296    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276300    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276305    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276310    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276315    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276320    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276325    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276329    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276334    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276347    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276353    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276358    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276362    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276367    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276375    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276381    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.281911 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276385    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276390    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276395    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276399    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276405    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276409    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276414    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276419    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276423    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276428    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276433    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276437    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276442    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276447    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276452    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276457    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276461    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276466    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276471    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.282402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276477    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276485    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276492    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276498    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276504    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276509    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276514    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276527    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276532    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276537    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276542    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276548    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276553    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276557    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276562    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276567    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276572    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276577    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276582    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276586    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.283245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276591    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276596    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276601    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276605    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276610    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276615    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276620    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276624    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276629    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276634    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276660    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276665    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276670    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276677    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276682    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276687    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276692    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276697    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276701    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276706    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.284128 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276719    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276724    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276730    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276737    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276742    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276746    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276751    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276756    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276761    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276765    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276770    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276775    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276779    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276784    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276789    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276794    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276798    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276803    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276808    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276812    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.285043 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276819    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276825    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.276831    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.276981    5534 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.276993    5534 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277013    5534 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277021    5534 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277053    5534 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277061    5534 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277068    5534 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277075    5534 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277080    5534 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277086    5534 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277101    5534 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277107    5534 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277115    5534 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277120    5534 flags.go:64] FLAG: --cgroup-root=""
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277125    5534 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277130    5534 flags.go:64] FLAG: --client-ca-file=""
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277135    5534 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277141    5534 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277152    5534 flags.go:64] FLAG: --cluster-domain=""
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277157    5534 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277162    5534 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 17:13:14.285981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277168    5534 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277174    5534 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277181    5534 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277186    5534 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277192    5534 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277197    5534 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277202    5534 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277208    5534 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277213    5534 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277218    5534 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277224    5534 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277231    5534 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277236    5534 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277244    5534 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277249    5534 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277254    5534 flags.go:64] FLAG: --enable-server="true"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277260    5534 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277270    5534 flags.go:64] FLAG: --event-burst="100"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277276    5534 flags.go:64] FLAG: --event-qps="50"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277281    5534 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277286    5534 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277292    5534 flags.go:64] FLAG: --eviction-hard=""
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277307    5534 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277315    5534 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277320    5534 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 17:13:14.287073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277325    5534 flags.go:64] FLAG: --eviction-soft=""
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277330    5534 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277335    5534 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277341    5534 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277346    5534 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277351    5534 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277357    5534 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277362    5534 flags.go:64] FLAG: --feature-gates=""
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277369    5534 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277374    5534 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277379    5534 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277385    5534 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277390    5534 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277396    5534 flags.go:64] FLAG: --help="false"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277401    5534 flags.go:64] FLAG: --hostname-override="ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277406    5534 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277412    5534 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277417    5534 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277423    5534 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277429    5534 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277437    5534 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277442    5534 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277447    5534 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 17:13:14.288172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277452    5534 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277458    5534 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277463    5534 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277468    5534 flags.go:64] FLAG: --kube-reserved=""
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277473    5534 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277479    5534 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277485    5534 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277500    5534 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277505    5534 flags.go:64] FLAG: --lock-file=""
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277510    5534 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277515    5534 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277521    5534 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277530    5534 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277536    5534 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277541    5534 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277546    5534 flags.go:64] FLAG: --logging-format="text"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277551    5534 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277557    5534 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277562    5534 flags.go:64] FLAG: --manifest-url=""
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277567    5534 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277575    5534 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277580    5534 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277586    5534 flags.go:64] FLAG: --max-pods="110"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277591    5534 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277597    5534 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 17:13:14.289244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277602    5534 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277607    5534 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277613    5534 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277618    5534 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277625    5534 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277636    5534 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277660    5534 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277665    5534 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277670    5534 flags.go:64] FLAG: --pod-cidr=""
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277675    5534 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277680    5534 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277686    5534 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277691    5534 flags.go:64] FLAG: --port="10250"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277696    5534 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277703    5534 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7b45970d80d2ce6"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277720    5534 flags.go:64] FLAG: --qos-reserved=""
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277725    5534 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277731    5534 flags.go:64] FLAG: --register-node="true"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277736    5534 flags.go:64] FLAG: --register-with-taints=""
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277743    5534 flags.go:64] FLAG: --registry-burst="10"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277748    5534 flags.go:64] FLAG: --registry-qps="5"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277753    5534 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277758    5534 flags.go:64] FLAG: --reserved-memory=""
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277765    5534 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277770    5534 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277775    5534 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 17:13:14.290330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277781    5534 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277786    5534 flags.go:64] FLAG: --runonce="false"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277791    5534 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277797    5534 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277802    5534 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277807    5534 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277813    5534 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277818    5534 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277824    5534 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277829    5534 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277835    5534 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277840    5534 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277845    5534 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277850    5534 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277855    5534 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277860    5534 flags.go:64] FLAG: --system-cgroups=""
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277865    5534 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277875    5534 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277879    5534 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277884    5534 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277899    5534 flags.go:64] FLAG: --tls-min-version=""
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277904    5534 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277917    5534 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277923    5534 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277928    5534 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 17:13:14.291479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277933    5534 flags.go:64] FLAG: --v="2"
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277940    5534 flags.go:64] FLAG: --version="false"
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277948    5534 flags.go:64] FLAG: --vmodule=""
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277954    5534 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.277960    5534 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278231    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278238    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278243    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278248    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278253    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278257    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278262    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278267    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278271    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278276    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278281    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278285    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278290    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278295    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278300    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.292547 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278304    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278309    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278315    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278320    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278324    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278329    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278334    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278341    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278346    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278351    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278357    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278362    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278366    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278371    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278376    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278380    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278385    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278390    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278394    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.293374 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278402    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278408    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278413    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278418    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278422    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278428    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278433    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278437    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278442    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278447    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278452    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278457    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278461    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278465    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278470    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278475    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278480    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278485    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278489    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.293942 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278494    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278500    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278507    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278513    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278518    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278532    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278537    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278542    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278547    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278560    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278565    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278570    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278574    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278580    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278584    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278589    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278594    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278599    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278603    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278608    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.294441 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278612    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278617    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278621    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278626    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278630    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278635    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278658    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278662    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278667    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278672    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278677    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278682    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278686    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278691    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278696    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278701    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278705    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278712    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278726    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278731    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.295091 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278736    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278741    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278746    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278751    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278755    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278760    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278765    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278770    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278774    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278779    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278784    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278789    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278794    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.278838    5534 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278917    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.295579 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278925    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278931    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278936    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278941    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278945    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278950    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278955    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278959    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278964    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278972    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278978    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278983    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278989    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.278994    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279000    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279006    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279020    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279025    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279030    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.295967 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279035    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279039    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279044    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279049    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279054    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279059    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279063    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279068    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279073    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279077    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279083    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279090    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279096    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279101    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279106    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279111    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279116    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279121    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279125    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.296483 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279130    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279135    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279139    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279153    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279158    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279163    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279168    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279172    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279177    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279189    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279195    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279208    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279214    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279218    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279223    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279227    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279232    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279237    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279242    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279257    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.297057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279262    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279266    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279271    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279276    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279280    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279285    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279290    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279295    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279300    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279304    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279309    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279313    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279318    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279323    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279328    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279333    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279338    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279342    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279347    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.297529 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279352    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279357    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279364    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279369    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279373    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279386    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279391    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279396    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279401    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279406    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279410    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279415    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279420    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279425    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279429    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279435    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279440    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279446    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279450    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279455    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.298161 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279460    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279465    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279471    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279475    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279480    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279485    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279489    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.279494    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.285134    5534 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.285153    5534 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.285178    5534 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.285186    5534 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285282    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285290    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285296    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285301    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285308    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285316    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285321    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.298673 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285326    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285331    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285336    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285341    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285345    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285350    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285355    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285360    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285365    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285370    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285374    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285379    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285384    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285391    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285398    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285405    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285410    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285415    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285419    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.299134 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285424    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285434    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285439    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285444    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285449    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285453    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285458    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285463    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285467    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285472    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285477    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285481    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285486    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285491    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285496    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285500    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285505    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285510    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285515    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.299763 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285519    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285533    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285538    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285543    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285548    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285552    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285557    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285562    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285566    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285571    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285576    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285581    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285585    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285589    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285594    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285599    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285605    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285610    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285615    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285619    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.300486 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285624    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285628    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285633    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285637    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285660    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285665    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285669    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285673    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285678    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285682    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285687    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285692    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285697    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285701    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285708    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285714    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285718    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285723    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285727    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.301080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285732    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285737    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285742    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285748    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285752    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285757    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285762    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285767    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285772    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285777    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285781    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285786    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285791    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285796    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285800    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285805    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285809    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285814    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285819    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285824    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.301537 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285828    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.285833    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.285876    5534 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286126    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286136    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286141    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286146    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286151    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286157    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286162    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286167    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286172    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286177    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286181    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286186    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286191    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.302033 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286196    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286201    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286205    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286211    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286218    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286223    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286227    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286232    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286237    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286242    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286247    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286252    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286256    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286261    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286266    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286270    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286275    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286280    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286285    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286290    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.302411 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286294    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286299    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286304    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286310    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286315    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286322    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286327    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286331    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286336    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286341    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286348    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286356    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286361    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286367    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286371    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286376    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286383    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286387    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286392    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.302969 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286397    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286402    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286407    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286412    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286417    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286422    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286427    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286431    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286436    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286441    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286446    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286450    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286456    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286461    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286466    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286470    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286478    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286483    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286488    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286494    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.303418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286499    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286504    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286509    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286514    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286519    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286524    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286528    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286533    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286538    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286545    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286549    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286563    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286568    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286573    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286577    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286585    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286593    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286598    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286603    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286608    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.303922 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286612    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286617    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286622    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286627    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286631    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286636    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286659    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286665    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286671    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286676    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286681    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286686    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286692    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286697    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.286736    5534 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 17:13:14.304456 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286789    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286796    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286802    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286806    5534 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286811    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286815    5534 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286820    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286829    5534 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286834    5534 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286839    5534 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286844    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286849    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286853    5534 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286858    5534 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286864    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286869    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286873    5534 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286878    5534 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286883    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286888    5534 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 17:13:14.304936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286893    5534 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286897    5534 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286902    5534 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286907    5534 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286912    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286919    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286924    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286929    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286933    5534 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286938    5534 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286944    5534 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286949    5534 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286954    5534 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286959    5534 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286964    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286969    5534 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286974    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286978    5534 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286983    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286990    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 17:13:14.305416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.286995    5534 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287000    5534 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287005    5534 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287010    5534 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287015    5534 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287020    5534 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287024    5534 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287029    5534 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287034    5534 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287041    5534 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287047    5534 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287053    5534 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287058    5534 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287063    5534 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287068    5534 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287072    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287077    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287084    5534 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287088    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 17:13:14.305918 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287093    5534 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287098    5534 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287102    5534 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287107    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287113    5534 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287118    5534 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287123    5534 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287128    5534 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287133    5534 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287139    5534 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287144    5534 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287149    5534 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287154    5534 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287159    5534 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287163    5534 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287168    5534 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287173    5534 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287178    5534 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287182    5534 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287187    5534 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 17:13:14.306418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287192    5534 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287197    5534 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287201    5534 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287207    5534 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287211    5534 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287216    5534 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287221    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287225    5534 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287230    5534 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287234    5534 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287241    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287246    5534 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287251    5534 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287255    5534 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287263    5534 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287270    5534 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287276    5534 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287282    5534 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287287    5534 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 17:13:14.306936 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287292    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287296    5534 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287301    5534 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287306    5534 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287311    5534 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287316    5534 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287321    5534 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:13:14.287325    5534 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.287402    5534 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.288946    5534 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.289027    5534 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.289403    5534 server.go:1014] "Starting client certificate rotation"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.289548    5534 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.289597    5534 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:07:03 +0000 UTC" deadline="2028-05-12 16:33:26.746147426 +0000 UTC"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.289636    5534 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13439h20m12.456514515s"
Oct 30 17:13:14.307402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.290163    5534 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 17:13:14.307856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.292024    5534 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 17:13:14.307856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.293845    5534 log.go:25] "Validated CRI v1 runtime API"
Oct 30 17:13:14.307856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.295660    5534 log.go:25] "Validated CRI v1 image API"
Oct 30 17:13:14.307856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.296337    5534 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 17:13:14.307856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.298701    5534 fs.go:135] Filesystem UUIDs: map[01203fc1-c7d6-4797-b511-f669187fe360:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 93c6ddc4-ffdf-416a-94af-e9f37a28134c:/dev/nvme0n1p4]
Oct 30 17:13:14.308012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.298721    5534 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee8cc6acf6a43ba59903ba132a699792b44857324f40679998426734d7c6ba89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee8cc6acf6a43ba59903ba132a699792b44857324f40679998426734d7c6ba89/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~projected/kube-api-access-wlt82:{mountpoint:/var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~projected/kube-api-access-wlt82 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ccfea3-7a41-4925-9669-c4a25efabc4a/volumes/kubernetes.io~projected/kube-api-access-hc79p:{mountpoint:/var/lib/kubelet/pods/34ccfea3-7a41-4925-9669-c4a25efabc4a/volumes/kubernetes.io~projected/kube-api-access-hc79p major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~projected/kube-api-access-rb8wh:{mountpoint:/var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~projected/kube-api-access-rb8wh major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/412abd52-f4da-491a-b694-30750614e3b4/volumes/kubernetes.io~projected/kube-api-access-x5lhx:{mountpoint:/var/lib/kubelet/pods/412abd52-f4da-491a-b694-30750614e3b4/volumes/kubernetes.io~projected/kube-api-access-x5lhx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~projected/kube-api-access-srxqx:{mountpoint:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~projected/kube-api-access-srxqx major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/622e2edd-70ee-4176-8498-50a740a59f82/volumes/kubernetes.io~projected/kube-api-access-4mzl9:{mountpoint:/var/lib/kubelet/pods/622e2edd-70ee-4176-8498-50a740a59f82/volumes/kubernetes.io~projected/kube-api-access-4mzl9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a1c6d38-094f-41a0-8c77-b1f54d231ea7/volumes/kubernetes.io~projected/kube-api-access-j6zv5:{mountpoint:/var/lib/kubelet/pods/6a1c6d38-094f-41a0-8c77-b1f54d231ea7/volumes/kubernetes.io~projected/kube-api-access-j6zv5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a3c7100-3ff5-4410-abd7-03e68c27baa3/volumes/kubernetes.io~projected/kube-api-access-sxwlt:{mountpoint:/var/lib/kubelet/pods/7a3c7100-3ff5-4410-abd7-03e68c27baa3/volumes/kubernetes.io~projected/kube-api-access-sxwlt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volumes/kubernetes.io~projected/kube-api-access-mlcdl:{mountpoint:/var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volumes/kubernetes.io~projected/kube-api-access-mlcdl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~projected/kube-api-access-6vhq7:{mountpoint:/var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~projected/kube-api-access-6vhq7 major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/991f66af-6bcb-4bd8-921f-58f2fb879067/volumes/kubernetes.io~projected/kube-api-access-fpshs:{mountpoint:/var/lib/kubelet/pods/991f66af-6bcb-4bd8-921f-58f2fb879067/volumes/kubernetes.io~projected/kube-api-access-fpshs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/config-reloader/2 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/prometheus/15 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~empty-dir/config-out major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~projected/kube-api-access-8gxr4:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~projected/kube-api-access-8gxr4 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~projected/tls-assets major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/web-config major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4/volumes/kubernetes.io~projected/kube-api-access-mltk9:{mountpoint:/var/lib/kubelet/pods/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4/volumes/kubernetes.io~projected/kube-api-access-mltk9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6a516e5-f016-4d54-be70-b1979fe0c605/volumes/kubernetes.io~projected/kube-api-access-tl9k9:{mountpoint:/var/lib/kubelet/pods/a6a516e5-f016-4d54-be70-b1979fe0c605/volumes/kubernetes.io~projected/kube-api-access-tl9k9 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6a516e5-f016-4d54-be70-b1979fe0c605/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a6a516e5-f016-4d54-be70-b1979fe0c605/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6b374b9-fd59-4579-8ebd-fe975116467c/volumes/kubernetes.io~projected/kube-api-access-9m4hz:{mountpoint:/var/lib/kubelet/pods/a6b374b9-fd59-4579-8ebd-fe975116467c/volumes/kubernetes.io~projected/kube-api-access-9m4hz major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6b374b9-fd59-4579-8ebd-fe975116467c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a6b374b9-fd59-4579-8ebd-fe975116467c/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~projected/kube-api-access-5ddxd:{mountpoint:/var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~projected/kube-api-access-5ddxd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa7322d-ff57-4460-be07-755ae180ffe4/volumes/kubernetes.io~projected/kube-api-access-g4zh6:{mountpoint:/var/lib/kubelet/pods/caa7322d-ff57-4460-be07-755ae180ffe4/volumes/kubernetes.io~projected/kube-api-access-g4zh6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa7322d-ff57-4460-be07-755ae180ffe4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/caa7322d-ff57-4460-be07-755ae180ffe4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a/volumes/kubernetes.io~projected/kube-api-access-drb5c:{mountpoint:/var/lib/kubelet/pods/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a/volumes/kubernetes.io~projected/kube-api-access-drb5c major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d255144b-ecb7-489b-98ca-a188acb6c109/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d255144b-ecb7-489b-98ca-a188acb6c109/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df1c926e-5a6b-4c44-881c-5dc47879a022/volumes/kubernetes.io~projected/kube-api-access-ccb7x:{mountpoint:/var/lib/kubelet/pods/df1c926e-5a6b-4c44-881c-5dc47879a022/volumes/kubernetes.io~projected/kube-api-access-ccb7x major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df1c926e-5a6b-4c44-881c-5dc47879a022/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/df1c926e-5a6b-4c44-881c-5dc47879a022/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:367 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/585c48b7958ea8abeb5e45dc9180ef7fc31899209ea6ee34c3e26e0277b7c996/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/05740831ba61ad021fa4eebcd6d8d481a2d8ce54cdcb5f6981588a1c70bc4b7e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2aca6a0abccf82e9cde84d9af4ac846eec57e9d88a54a62ed64df276b0041b36/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b92b3ef6f9f78a4c45bb6031a1267ca76b896965bb3bd968d24c24487db617bf/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/488b9f49fd13111aa43eafc82de251cac4da53afab6bd0a2f956f1ee1e08c64e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ba70276cbb1e4c786657348e22c660bbabc28438609bb5ee7a75b97be1e58fbe/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2b2c6ead619759f62aaf149c0e40505a943172a918ece9d98adaed11beec7127/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/150fcccde14a2acac7af6e54570d07620473035cbec6433a91fb783bd23b28f5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/515d2b3aecccfa04eacc9e51e9f1e0f363aae3ae126287fd9f3ed6b7349972b9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f76517e2a529bf1b61aaf6a52d7a15fd4f01a9d98863c2c856e3f42051077a8c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7ba89c5ddc46befcc21579d5e488b99ff7eab1a13ef501286f861daef376ddf2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/8ee242b15d79a71ff2e16ab790f1fe99598f863afe80553eea684a83b86bce80/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e9fea9ff079824b80e67d2b54d6f58fbb57de6e2aa6618f3c5dd3ad9dc611fd9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/73391dcbf70355d5d0c11d6b1fde3a916a5564b49b5d8487f94d7482cbae441a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4b09de495bac96f7efd490d7cfa3153601ffbebdabfbc0fa453678e66562bef1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1c93262a047a0dae8215442c6ca4a9717347da3e077476a588c865145a1ab339/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6c993af85c14d58f692fe087ee4d320d5cb8ab82874986e4c1cbd757d13d9fc3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/75a7d2ce3c5c932ca5481e290af6f7aa3e5bbf7798b76500d31f0797a5091a47/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8d20f9bc6e768f03f82db33559a9be3149ba4c7f12ac4b33f52a39337e42b16a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0112aace330a82f882b7d00bb06d8526af2e3ba1d92e28af3888dfe5e67fd6d2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/358c7d956fea53d43fb21e1ac34ca9030d8ee2be508061e3873768f2707b6cfc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a4b0b5a52539ebddb874701e71b5537895ba6c376c7a32f049f1e6eaf0817587/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/73da99ba7a56152328c5e983a01d9f237fac905a5a1f35bd6dd49231197af70d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/80f2ee81e64607f661201a4deebf314eab83f9dd1a97c48004b310f122aa74ac/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/8480d8feca43c842f4ae37a21886b4b9e4082476558eb1066f31f7618fbee510/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/caac200aa8cc9aabdfc9f322dbc3da6fe3f3f1e7a09cd4d4723bb7f770010635/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/1b5df5e9096d4c12b3b934c83ed68668f3bbba74bd56ebec773e1034a69f5dd4/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d68b8e6ba769126c0a2c3d9a5c33e437156b60f0c3e4f7736b6d6e6998f551cb/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/f4cc46d4c1b2c05e2bbdd9cfe0522da590cd4b0b2f31c0bfce2c63246a907f1e/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e99be8ab5bf74f5e27446fd2269f12e800b7443f3f99ebbe9736c998f6745916/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/ecdc7a43886061a71582418da29a8686cd02643208a5c9b5574026ce9ce861d6/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/04e5c0f6bcac14222e57193d326dadbddd1190a2296d9b57249d8df9893304df/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/35645b5ceda19bb19d4cb2520efa39bcc960a63d6c76639b426879e4d99724eb/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/342eaa1b70f941513e2179372a60efd5552f84b7002f5254c42edeae1bcdf2f1/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b4152f2dfe5ef7b4db23023d452ae7f0cf9e8f93c8b85b3ca06a6f2dde8a3b7e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/7794ef427d238cbed52cc7810752d7be909614c102845e261174c34ed615b512/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/8376f6865ad31c6fbcb1953dc34fbcc0c54ee2be3c1ed9f7575393c158d87d46/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/2ad1410f386c768c1b777492f6b848c57689079f91307db06c83ee8213d3436e/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/173292b02f86e5b0667ecfd1f739941771f91b7289bb7ed13bfcf49f5397b4a9/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/6c8a7f069d91fac2a60e2b295d609ae4500c4158bfcd48a684392a470e8c79f4/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/93aa5e89faf34500900c92f78703fa4f15d1148ee9a9efc72b689f6ad8dd042c/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/2b6583341e09b9965322b4e470a8c8e294b75e190ac087089e1980b0b1e5e99e/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/68fc1dc27864bb19f2a8fc3f091474c10faddf183f99d8dd09955eee2e540264/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/dcf1707a7187f36e7b38abb38efd8c5b8463d71c3a8805495ab52b280463f60e/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/612d233d628e0afe6e6f4101eb78acd630773b54b6c6924f2f25e95a35862765/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/9e334a7db52b82e22d9e890b497f3e33325d1711031080423e3c4d773406c3ad/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a3dcbd5f6020937fda9518a9125070c522abc8b48024a02c042d0a054c5ae211/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/b6a786463f03f1c17d750b243a0ae92650b6166fd2a3b6a2a1cd50db6d4e0efd/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/e1d7b4188bf8793e863816a9d6e3e76fd9a2d85da92bde78a9a19aa567ffbf00/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/82e65c83118b6c6965eaf9aa8de71c2f257ff683ac9594334b78d3f926c08007/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/fdf56f7173e9f553ab7b17f51a54a2d155d878fe892627ab24947c9b1d212102/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/c1012fa44d4f09f526bfcde97b11f31a79ae4947d341280522fb5fc9b39d5972/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/b9f738d1b031c8391c87c0e8c061d3d59bea8e477613a93a58bf91673338afd0/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/c09206325d9226c6631e22384b97a9f4bf6edc8f2811899bf83d6c77bad7002a/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/b412a0827008215901205831c86baf28c3eacffe65a1c671fe3e00eb441f82d2/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/0b08f949c14f7d0d1c7edf4115e9401597ab97ab00ac274de10b846d551d07a0/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/1f824c4c1cb0d5ae753a3ba13ec647642cd70fc47a7a62b4250cc3eb57c55c87/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/397bdac91bef0f6e58af9afc8c29465b9f18742fc13b1d7384066232d827738c/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/b25573dfbaa675c9abdcbae17a83442fe399405e9314c0bcba63d20897041e17/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/b26116df32700298b492e5844696648109756b0fb949816799ea30a00f089989/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/caa8a546818672379133aa3b3876b4d451b3ed3ea057d3355b8741c21c81cce3/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5b736b5c76030aec1bf7b0ed6c4ada637bd4e0d36d53b4ec89a901e81e4e5154/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9d2cc462ae7922f26aa55c5055718af103591279c998021d27ed84fe21750204/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/870108ffd02b1243e1534bf8d9e66b7d3a40b0248cd21c9f5cdff2a308991253/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/849158a196f2ffe08002a3a563e5462c587f80a394f7cc4c13a0de4caf77f3b7/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/49e696b3cc029cc8e5a478d2a816c824ae17587af6fd7498fea54248162bf43f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ced2bb36804282273896a62541f52644d939a49fcf23ad7ea970dc11247f6873/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ca60a8994cd3555115d673483ba01669b34a3e5449d02968c2e22c169136cc8d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1773889a239b0938a69c5d8666aaf501f5634b2fd07218191986b756e6662965/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/108106cbfbd7d52fbe3445923fe7dd86ad94ddc09199166deee9fc8a4624c8aa/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9186e84954616557f533bd7f902412a10a18f22e4c7b3eb3d01c5d3a11a177b7/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.306841    5534 manager.go:217] Machine: {Timestamp:2026-10-30 17:13:14.306352323 +0000 UTC m=+0.068396127 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126331 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fb86fcbd27ce842402ef800352681 SystemUUID:ec2fb86f-cbd2-7ce8-4240-2ef800352681 BootID:5527dc6b-d434-4c49-baf2-36e2e7458972 Filesystems:[{Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/029563d6fad0db4271473195a10202439b251fa9052cf07a9d8e78a4a0f1d82a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff0f5ee6ba06dcefb14ca14bf5bb4542cc7ccc12f0857985165d346e5b0ccf5a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caa7322d-ff57-4460-be07-755ae180ffe4/volumes/kubernetes.io~projected/kube-api-access-g4zh6 DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd5e2440522f886965e8cd2bca4909409c9b2ab0a8ab6f3ce6a1424a2705cfe4/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ed964ca4e0f61a1dde38364fd01a92446d6dee37d5c73c1f8b4a721cfb3bff4/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c616e1189d6eb1fb68010ee5047f87014a6fd8a436b1720a57b4b9def94cf68e/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f279fc9fb2279a2ae36a5273eef186214b29064f714b6f6e7de0cd9a1479bc9/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~projected/kube-api-access-8gxr4 DeviceMajor:0 DeviceMinor:345 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a6b374b9-fd59-4579-8ebd-fe975116467c/volumes/kubernetes.io~projected/kube-api-access-9m4hz DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cb219a819bec29f219f4c73693c6f1add110f5cfc57f5c527abbac24944acf0/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:363 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa7a488a1fd519001b87ef5613d861dae27d6c2759bb86bdee56ae0c2e8b9249/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~projected/kube-api-access-6vhq7 DeviceMajor:0 DeviceMinor:257 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~projected/kube-api-access-rb8wh DeviceMajor:0 DeviceMinor:252 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df1c926e-5a6b-4c44-881c-5dc47879a022/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:310 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~projected/kube-api-access-5ddxd DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd24dca78d82a19f9fac894a45b820993e0ddda37edc01bb5e8b4e342a282c82/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:322 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee8cc6acf6a43ba59903ba132a699792b44857324f40679998426734d7c6ba89/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:367 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6a1c6d38-094f-41a0-8c77-b1f54d231ea7/volumes/kubernetes.io~projected/kube-api-access-j6zv5 DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6b374b9-fd59-4579-8ebd-fe975116467c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:256 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~projected/kube-api-access-wlt82 DeviceMajor:0 DeviceMinor:258 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:340 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caa54cca-be01-4afd-8690-5cb0958c1027/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/412abd52-f4da-491a-b694-30750614e3b4/volumes/kubernetes.io~projected/kube-api-access-x5lhx DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:337 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:343 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4/volumes/kubernetes.io~projected/kube-api-access-mltk9 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84532c65-2d04-45a5-b0cb-9841d8e20d87/volumes/kubernetes.io~projected/kube-api-access-mlcdl DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df1c926e-5a6b-4c44-881c-5dc47879a022/volumes/kubernetes.io~projected/kube-api-access-ccb7x DeviceMajor:0 DeviceMinor:314 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d98fbd8c87046c8503b7c1bf77969dfd6cfdc441b65c798b68c54ee51ceeccba/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:338 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:344 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/322b7165-7398-4ed9-99a9-cef5e8ee3c86/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/908859a0-1e77-4a62-a253-3486242475d8/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a3c7100-3ff5-4410-abd7-03e68c27baa3/volumes/kubernetes.io~projected/kube-api-access-sxwlt DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c52cdc26b73b51d4e868d8cc342075bcb25e75cdd0e6cb1afc0043801f0866cf/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7d7a398169422ebeffbd3c0e1cfbb8cb6e3c2f7d7c6a3011dcf746a9dd3ab38/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/991f66af-6bcb-4bd8-921f-58f2fb879067/volumes/kubernetes.io~projected/kube-api-access-fpshs DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:320 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6a516e5-f016-4d54-be70-b1979fe0c605/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a6a516e5-f016-4d54-be70-b1979fe0c605/volumes/kubernetes.io~projected/kube-api-access-tl9k9 DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/376a21c0-10de-43a4-a04c-c3519a868a9b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:341 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/34ccfea3-7a41-4925-9669-c4a25efabc4a/volumes/kubernetes.io~projected/kube-api-access-hc79p DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:321 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:342 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff3884c35cc80dcb6db1e5d169e85e7d669fa2ca66f085648a53ebf2fffd84cc/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9dd49d55124da5e263e6a3053de2e49a67f49cc3c5a75cc141dd8fd11472cc2/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3896b91f3c07a4cd7ac8867d8580bd777d93355fd72dea5285478ececb29cb8b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d292bde84b4a1b0bf88efbfe34a1262ab0fd63cc53f2817b2a26f30124bc4dff/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caa7322d-ff57-4460-be07-755ae180ffe4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a/volumes/kubernetes.io~projected/kube-api-access-drb5c DeviceMajor:0 DeviceMinor:368 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c86d1f6-23f6-4d70-bcac-400d7c08bc09/volumes/kubernetes.io~projected/kube-api-access-srxqx DeviceMajor:0 DeviceMinor:323 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9dca0b75-8ca5-4e65-8c40-75ca747fffb3/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:335 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd2e75e5491b4c4e4a2813354ddb7076dd66edb88645c99439a907038f123cdd/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d255144b-ecb7-489b-98ca-a188acb6c109/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/622e2edd-70ee-4176-8498-50a740a59f82/volumes/kubernetes.io~projected/kube-api-access-4mzl9 DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a54c9d2fb8ff4d9327fd26695b9e963df318758b0e709436c09890769b44c1d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6be8fc8a39312f71a5fa170e887627dcf7a9ceb59a6a382fb1887164d9aa7f4/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:029563d6fad0db4 MacAddress:ca:28:21:1e:9f:da Speed:10000 Mtu:8901} {Name:3f279fc9fb2279a MacAddress:e6:14:79:2f:7f:cc Speed:10000 Mtu:8901} {Name:4ed964ca4e0f61a MacAddress:26:00:cd:42:f0:a6 Speed:10000 Mtu:8901} {Name:8cb219a819bec29 MacAddress:12:ae:48:59:9c:89 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c2:c6:86:29 Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:9a:e6:31:2c:9e Speed:0 Mtu:8901} {Name:c616e1189d6eb1f MacAddress:b2:3d:84:22:1d:86 Speed:10000 Mtu:8901} {Name:d292bde84b4a1b0 MacAddress:9e:4a:cd:8f:5c:9a Speed:10000 Mtu:8901} {Name:d98fbd8c87046c8 MacAddress:fe:fb:ce:0a:aa:c3 Speed:10000 Mtu:8901} {Name:dd2e75e5491b4c4 MacAddress:76:56:80:24:5a:7c Speed:10000 Mtu:8901} {Name:e6be8fc8a39312f MacAddress:be:69:33:18:d9:a4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c2:c6:86:29 Speed:0 Mtu:9001} {Name:fd5e2440522f886 MacAddress:b2:37:aa:87:ae:dc Speed:10000 Mtu:8901} {Name:ff3884c35cc80dc MacAddress:d2:2b:80:f7:0d:d2 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e6:e9:4f:8f:94:fd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:f8:28:ce:02:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307210    5534 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307254    5534 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307543    5534 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307577    5534 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307754    5534 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307762    5534 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307772    5534 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307791    5534 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.307885    5534 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308049    5534 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308150    5534 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308182    5534 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308192    5534 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308207    5534 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308215    5534 kubelet.go:405] "Adding apiserver pod source"
Oct 30 17:13:14.308224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308225    5534 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 17:13:14.309000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.308986    5534 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 17:13:14.309000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.309001    5534 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 17:13:14.309509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.309492    5534 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 17:13:14.311977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.311963    5534 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 17:13:14.311977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.311977    5534 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 17:13:14.312087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312077    5534 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 17:13:14.312661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312633    5534 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 17:13:14.312700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312681    5534 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 17:13:14.312789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312780    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 17:13:14.312819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312795    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 17:13:14.312819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312802    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 17:13:14.312819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312807    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 17:13:14.312819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312815    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312826    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312835    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312841    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312848    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312853    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312868    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312878    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312906    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 17:13:14.312922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.312915    5534 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 17:13:14.316315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.316304    5534 server.go:1263] "Started kubelet"
Oct 30 17:13:14.316410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.316373    5534 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 17:13:14.316943 ip-10-0-135-170 systemd[1]: Started Kubernetes Kubelet.
Oct 30 17:13:14.317161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.316929    5534 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 17:13:14.317161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.317000    5534 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 17:13:14.317776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.317740    5534 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 17:13:14.319513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.319494    5534 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 17:13:14.322907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.322876    5534 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 17:13:14.322907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.322899    5534 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 17:13:14.323084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.322952    5534 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:07:04 +0000 UTC" deadline="2028-08-16 13:08:39.140749408 +0000 UTC"
Oct 30 17:13:14.323084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.322974    5534 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15739h55m24.817780346s"
Oct 30 17:13:14.323084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.323001    5534 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 17:13:14.323084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.323014    5534 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 17:13:14.323084 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.323010    5534 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-170.ec2.internal\" not found"
Oct 30 17:13:14.323316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.323206    5534 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 17:13:14.325072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.325044    5534 factory.go:55] Registering systemd factory
Oct 30 17:13:14.325072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.325072    5534 factory.go:223] Registration of the systemd container factory successfully
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.325993    5534 factory.go:153] Registering CRI-O factory
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326013    5534 factory.go:223] Registration of the crio container factory successfully
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326039    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-client-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326070    5534 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326076    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326088    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84532c65-2d04-45a5-b0cb-9841d8e20d87" volumeName="kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326093    5534 factory.go:103] Registering Raw factory
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326097    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa7322d-ff57-4460-be07-755ae180ffe4" volumeName="kubernetes.io/empty-dir/caa7322d-ff57-4460-be07-755ae180ffe4-data-volume" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326107    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df1c926e-5a6b-4c44-881c-5dc47879a022" volumeName="kubernetes.io/secret/df1c926e-5a6b-4c44-881c-5dc47879a022-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326112    5534 manager.go:1199] Started watching for new ooms in manager
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326118    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-kube-api-access-8gxr4" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326128    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a1c6d38-094f-41a0-8c77-b1f54d231ea7" volumeName="kubernetes.io/projected/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kube-api-access-j6zv5" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326136    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322b7165-7398-4ed9-99a9-cef5e8ee3c86" volumeName="kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326146    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322b7165-7398-4ed9-99a9-cef5e8ee3c86" volumeName="kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-metrics-client-ca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326154    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908859a0-1e77-4a62-a253-3486242475d8" volumeName="kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326162    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d255144b-ecb7-489b-98ca-a188acb6c109" volumeName="kubernetes.io/configmap/d255144b-ecb7-489b-98ca-a188acb6c109-konnectivity-ca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326177    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="622e2edd-70ee-4176-8498-50a740a59f82" volumeName="kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-cni-binary-copy" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326192    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326203    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="622e2edd-70ee-4176-8498-50a740a59f82" volumeName="kubernetes.io/projected/622e2edd-70ee-4176-8498-50a740a59f82-kube-api-access-4mzl9" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326213    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="991f66af-6bcb-4bd8-921f-58f2fb879067" volumeName="kubernetes.io/projected/991f66af-6bcb-4bd8-921f-58f2fb879067-kube-api-access-fpshs" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326221    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6a516e5-f016-4d54-be70-b1979fe0c605" volumeName="kubernetes.io/secret/a6a516e5-f016-4d54-be70-b1979fe0c605-metrics-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326228    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322b7165-7398-4ed9-99a9-cef5e8ee3c86" volumeName="kubernetes.io/projected/322b7165-7398-4ed9-99a9-cef5e8ee3c86-kube-api-access-wlt82" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326235    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3c7100-3ff5-4410-abd7-03e68c27baa3" volumeName="kubernetes.io/configmap/7a3c7100-3ff5-4410-abd7-03e68c27baa3-serviceca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326243    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84532c65-2d04-45a5-b0cb-9841d8e20d87" volumeName="kubernetes.io/secret/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326250    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326257    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="376a21c0-10de-43a4-a04c-c3519a868a9b" volumeName="kubernetes.io/projected/376a21c0-10de-43a4-a04c-c3519a868a9b-kube-api-access-rb8wh" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326265    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="412abd52-f4da-491a-b694-30750614e3b4" volumeName="kubernetes.io/configmap/412abd52-f4da-491a-b694-30750614e3b4-iptables-alerter-script" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326274    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6a516e5-f016-4d54-be70-b1979fe0c605" volumeName="kubernetes.io/configmap/a6a516e5-f016-4d54-be70-b1979fe0c605-config-volume" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326283    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ccfea3-7a41-4925-9669-c4a25efabc4a" volumeName="kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326291    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="376a21c0-10de-43a4-a04c-c3519a868a9b" volumeName="kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326300    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326307    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326314    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326329    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a" volumeName="kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326340    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="376a21c0-10de-43a4-a04c-c3519a868a9b" volumeName="kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326352    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326364    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6a516e5-f016-4d54-be70-b1979fe0c605" volumeName="kubernetes.io/projected/a6a516e5-f016-4d54-be70-b1979fe0c605-kube-api-access-tl9k9" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326371    5534 manager.go:319] Starting recovery of all containers
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326376    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6b374b9-fd59-4579-8ebd-fe975116467c" volumeName="kubernetes.io/projected/a6b374b9-fd59-4579-8ebd-fe975116467c-kube-api-access-9m4hz" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326387    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322b7165-7398-4ed9-99a9-cef5e8ee3c86" volumeName="kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326398    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/projected/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-kube-api-access-srxqx" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326409    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326421    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3c7100-3ff5-4410-abd7-03e68c27baa3" volumeName="kubernetes.io/projected/7a3c7100-3ff5-4410-abd7-03e68c27baa3-kube-api-access-sxwlt" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326431    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="991f66af-6bcb-4bd8-921f-58f2fb879067" volumeName="kubernetes.io/empty-dir/991f66af-6bcb-4bd8-921f-58f2fb879067-tmp-dir" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326442    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326453    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df1c926e-5a6b-4c44-881c-5dc47879a022" volumeName="kubernetes.io/projected/df1c926e-5a6b-4c44-881c-5dc47879a022-kube-api-access-ccb7x" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326464    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326477    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326490    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326501    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326512    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326527    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4" volumeName="kubernetes.io/projected/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-kube-api-access-mltk9" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326539    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa54cca-be01-4afd-8690-5cb0958c1027" volumeName="kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-etc-tuned" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326550    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa7322d-ff57-4460-be07-755ae180ffe4" volumeName="kubernetes.io/projected/caa7322d-ff57-4460-be07-755ae180ffe4-kube-api-access-g4zh6" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326569    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ccfea3-7a41-4925-9669-c4a25efabc4a" volumeName="kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326581    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="376a21c0-10de-43a4-a04c-c3519a868a9b" volumeName="kubernetes.io/configmap/376a21c0-10de-43a4-a04c-c3519a868a9b-metrics-client-ca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326593    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-metrics-client-ca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326604    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908859a0-1e77-4a62-a253-3486242475d8" volumeName="kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326615    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326627    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-grpc-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326656    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3dcc262-c93f-43f7-ba5f-601ac5ab06b3" volumeName="kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326668    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322b7165-7398-4ed9-99a9-cef5e8ee3c86" volumeName="kubernetes.io/empty-dir/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-textfile" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326691    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322b7165-7398-4ed9-99a9-cef5e8ee3c86" volumeName="kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326703    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ccfea3-7a41-4925-9669-c4a25efabc4a" volumeName="kubernetes.io/projected/34ccfea3-7a41-4925-9669-c4a25efabc4a-kube-api-access-hc79p" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326714    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="622e2edd-70ee-4176-8498-50a740a59f82" volumeName="kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-multus-daemon-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326725    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84532c65-2d04-45a5-b0cb-9841d8e20d87" volumeName="kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-env-overrides" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326740    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908859a0-1e77-4a62-a253-3486242475d8" volumeName="kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-metrics-client-ca" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326751    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config-out" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326775    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-web-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326786    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ccfea3-7a41-4925-9669-c4a25efabc4a" volumeName="kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-binary-copy" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326797    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908859a0-1e77-4a62-a253-3486242475d8" volumeName="kubernetes.io/empty-dir/908859a0-1e77-4a62-a253-3486242475d8-volume-directive-shadow" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326809    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908859a0-1e77-4a62-a253-3486242475d8" volumeName="kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326821    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-tls-assets" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326833    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a1e416-7f8a-4940-85c2-19f0fd39c5f4" volumeName="kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326845    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6a516e5-f016-4d54-be70-b1979fe0c605" volumeName="kubernetes.io/empty-dir/a6a516e5-f016-4d54-be70-b1979fe0c605-tmp-dir" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326856    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6b374b9-fd59-4579-8ebd-fe975116467c" volumeName="kubernetes.io/secret/a6b374b9-fd59-4579-8ebd-fe975116467c-cert" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326867    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa54cca-be01-4afd-8690-5cb0958c1027" volumeName="kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-tmp" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326878    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="412abd52-f4da-491a-b694-30750614e3b4" volumeName="kubernetes.io/projected/412abd52-f4da-491a-b694-30750614e3b4-kube-api-access-x5lhx" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326892    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c86d1f6-23f6-4d70-bcac-400d7c08bc09" volumeName="kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-federate-client-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326903    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326913    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326924    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa54cca-be01-4afd-8690-5cb0958c1027" volumeName="kubernetes.io/projected/caa54cca-be01-4afd-8690-5cb0958c1027-kube-api-access-5ddxd" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326934    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908859a0-1e77-4a62-a253-3486242475d8" volumeName="kubernetes.io/projected/908859a0-1e77-4a62-a253-3486242475d8-kube-api-access-6vhq7" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326950    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326964    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa7322d-ff57-4460-be07-755ae180ffe4" volumeName="kubernetes.io/secret/caa7322d-ff57-4460-be07-755ae180ffe4-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326975    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d255144b-ecb7-489b-98ca-a188acb6c109" volumeName="kubernetes.io/secret/d255144b-ecb7-489b-98ca-a188acb6c109-agent-certs" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326982    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84532c65-2d04-45a5-b0cb-9841d8e20d87" volumeName="kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326990    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84532c65-2d04-45a5-b0cb-9841d8e20d87" volumeName="kubernetes.io/projected/84532c65-2d04-45a5-b0cb-9841d8e20d87-kube-api-access-mlcdl" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.326997    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" volumeName="kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.327004    5534 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa7322d-ff57-4460-be07-755ae180ffe4" volumeName="kubernetes.io/configmap/caa7322d-ff57-4460-be07-755ae180ffe4-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.327012    5534 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 17:13:14.329667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.327017    5534 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 17:13:14.359802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.359732    5534 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 17:13:14.361458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.361423    5534 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 17:13:14.361458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.361446    5534 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 17:13:14.361631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.361482    5534 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 17:13:14.361631 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.361523    5534 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 17:13:14.373040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.373011    5534 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:13:14.373171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.373150    5534 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:13:14.373370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.373349    5534 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 17:13:14.384247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.384219    5534 generic.go:356] "Generic (PLEG): container finished" podID="9dca0b75-8ca5-4e65-8c40-75ca747fffb3" containerID="875df2990e863457c688117c7ab8168564010d44d0b5990e32141b46af0fa6df" exitCode=0
Oct 30 17:13:14.389248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.389216    5534 generic.go:356] "Generic (PLEG): container finished" podID="322b7165-7398-4ed9-99a9-cef5e8ee3c86" containerID="0114a4c56a9ee68353246b697f60d4a82f8224cdc7350ea64a88a24eed10602c" exitCode=0
Oct 30 17:13:14.392861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.392828    5534 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="475670f469f57f31dacb73ce4ba279d122db8f93dc9a89a3a1351dcf33e41a26" exitCode=0
Oct 30 17:13:14.392925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.392867    5534 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="841b718a9a3562f7985f8c7167653a675d00eea9a8d444711dd72e93b6620bc6" exitCode=0
Oct 30 17:13:14.392925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.392875    5534 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="b1f61f3376b8693f2014426681afdda90e777fb24750d53245b11eb9ead48ca8" exitCode=0
Oct 30 17:13:14.392925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.392894    5534 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="7683259cfd3132ad3c313310dbdf7114c0d51786eebd25197d82695ce98e11b5" exitCode=0
Oct 30 17:13:14.392925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.392910    5534 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="b42b2114e954bcd0c8a0183b56d7c1caa7c788ef4d8d52a245a2f7496157451b" exitCode=0
Oct 30 17:13:14.392925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.392920    5534 generic.go:356] "Generic (PLEG): container finished" podID="34ccfea3-7a41-4925-9669-c4a25efabc4a" containerID="09b32b1fcdfbe4c55fd9afa3c7c4e924e6c15bf6aa60c1cfab3cc08a5e7f8ec9" exitCode=0
Oct 30 17:13:14.397038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.397018    5534 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:13:14.397115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.397084    5534 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 17:13:14.399323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.399304    5534 generic.go:356] "Generic (PLEG): container finished" podID="e477122cbf7bd238eb683471d99d303b" containerID="1982b1ac3d580ba1bc1ba397561f5bde56e2f1b61a25953809694dcb9fa11c80" exitCode=0
Oct 30 17:13:14.435957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.435909    5534 manager.go:324] Recovery completed
Oct 30 17:13:14.459277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459254    5534 cpu_manager.go:226] "Starting" policy="none"
Oct 30 17:13:14.459277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459271    5534 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 17:13:14.459453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459291    5534 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 17:13:14.459453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459444    5534 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 17:13:14.459551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459459    5534 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 17:13:14.459551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459486    5534 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 17:13:14.459551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459494    5534 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 17:13:14.459551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459502    5534 policy_none.go:50] "Start"
Oct 30 17:13:14.459551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459514    5534 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 17:13:14.459551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459528    5534 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 17:13:14.459843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459725    5534 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 17:13:14.459843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459736    5534 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 17:13:14.459843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.459743    5534 policy_none.go:44] "Start"
Oct 30 17:13:14.462582 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.462550    5534 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 17:13:14.466495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.466455    5534 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.473172    5534 manager.go:341] "Starting Device Plugin manager"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.473200    5534 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.473887    5534 server.go:87] "Starting device plugin registration server"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474214    5534 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474225    5534 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474617    5534 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474721    5534 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474753    5534 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474759    5534 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 17:13:14.476556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.474827    5534 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:13:14.474745026Z","UUID":"a241e1e6-dc08-410d-85ee-7a31d49643a9","Handler":null,"Name":"","Endpoint":""}
Oct 30 17:13:14.477180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.477157    5534 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 17:13:14.477240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.477202    5534 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 17:13:14.478623 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.478605    5534 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 17:13:14.575011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.574981    5534 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 17:13:14.576787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.576773    5534 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 17:13:14.576849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.576800    5534 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 17:13:14.576849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.576809    5534 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-170.ec2.internal" event="NodeHasSufficientPID"
Oct 30 17:13:14.576904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.576852    5534 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.610515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.610501    5534 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.610579 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.610550    5534 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.663592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.663543    5534 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"]
Oct 30 17:13:14.664272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664259    5534 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.664347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664309    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" event={"ID":"e477122cbf7bd238eb683471d99d303b","Type":"ContainerStarted","Data":"1d85f8985251b8c3667d484acb78c7699ab3ae530ac0037c40744095ff0a97fb"}
Oct 30 17:13:14.664378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664362    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" event={"ID":"e477122cbf7bd238eb683471d99d303b","Type":"ContainerDied","Data":"1982b1ac3d580ba1bc1ba397561f5bde56e2f1b61a25953809694dcb9fa11c80"}
Oct 30 17:13:14.664406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664391    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal" event={"ID":"e477122cbf7bd238eb683471d99d303b","Type":"ContainerStarted","Data":"84639020c4bb4230c98e35f90729709dc13d44b36504eb8a305b51a113f5cd43"}
Oct 30 17:13:14.664471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664459    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" event={"ID":"cd29d125f430f9e0a8a7b7b6580a8dec","Type":"ContainerStarted","Data":"9b60889084568a91f326fabbc49d493365b0f617afe55d460797520807b23e4f"}
Oct 30 17:13:14.664504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664474    5534 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.664504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.664483    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" event={"ID":"cd29d125f430f9e0a8a7b7b6580a8dec","Type":"ContainerStarted","Data":"b2ec7419ecd1c5ea0393472d29eb8e0ce0f9d4a4a84a86df7b28288fb0d364ee"}
Oct 30 17:13:14.687671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.687615    5534 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 17:13:14.687671 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.687665    5534 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.688072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.688060    5534 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 17:13:14.688107 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:14.688100    5534 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.777548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.777499    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.777693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.777563    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.777806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.777610    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd29d125f430f9e0a8a7b7b6580a8dec-config\") pod \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" (UID: \"cd29d125f430f9e0a8a7b7b6580a8dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.878948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.878921    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.879079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.878885    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.879079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.878996    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.879079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.879018    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd29d125f430f9e0a8a7b7b6580a8dec-config\") pod \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" (UID: \"cd29d125f430f9e0a8a7b7b6580a8dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.879079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.879053    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd29d125f430f9e0a8a7b7b6580a8dec-config\") pod \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" (UID: \"cd29d125f430f9e0a8a7b7b6580a8dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:14.879244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:14.879078    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e477122cbf7bd238eb683471d99d303b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" (UID: \"e477122cbf7bd238eb683471d99d303b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:15.309309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.309273    5534 apiserver.go:52] "Watching apiserver"
Oct 30 17:13:15.360694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.360664    5534 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 17:13:15.361301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.361277    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rcllv","openshift-monitoring/telemeter-client-85f86dfd46-lsb6t","openshift-ingress-canary/ingress-canary-pwswg","openshift-multus/multus-7k5d2","openshift-multus/network-metrics-daemon-dqkvr","kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-4hzkr","openshift-cluster-node-tuning-operator/tuned-nc4z8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw","openshift-network-diagnostics/network-check-target-z2hmf","kube-system/global-pull-secret-syncer-v8rgj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg","openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l","openshift-image-registry/node-ca-xjh5h","openshift-monitoring/node-exporter-mbfgv","openshift-multus/multus-additional-cni-plugins-5xbwb","openshift-dns/node-resolver-zz6f9","openshift-network-operator/iptables-alerter-6nccg","openshift-ovn-kubernetes/ovnkube-node-96dfd","openshift-insights/insights-runtime-extractor-p274z"]
Oct 30 17:13:15.376789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376765    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376810    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376850    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376881    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376897    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376917    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn7sb\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376928    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.376974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376973    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctbv7\""
Oct 30 17:13:15.377231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.376991    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.377231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.377015    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.377231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.377057    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 17:13:15.377231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.377088    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 17:13:15.377231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.377118    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8szd\""
Oct 30 17:13:15.386984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.386968    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lrx5\""
Oct 30 17:13:15.387183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387163    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2wbm5\""
Oct 30 17:13:15.387183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387174    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\""
Oct 30 17:13:15.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387204    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g55vh\""
Oct 30 17:13:15.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387287    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49f6v\""
Oct 30 17:13:15.387430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387418    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x2mz9\""
Oct 30 17:13:15.387472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387462    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vxcg\""
Oct 30 17:13:15.387891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387879    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 17:13:15.387929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387900    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 17:13:15.387971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.387960    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 17:13:15.388386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388375    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.388460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388447    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.388497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388462    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 17:13:15.388537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388493    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.388537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388511    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.388621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388609    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 17:13:15.388768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388757    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-726zt\""
Oct 30 17:13:15.388888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.388877    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zkp55\""
Oct 30 17:13:15.389153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.389138    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 17:13:15.389242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.389225    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\""
Oct 30 17:13:15.389309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.389298    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 17:13:15.389348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.389302    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w2nmx\""
Oct 30 17:13:15.389348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.389337    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 17:13:15.389348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.389344    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 17:13:15.390115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390101    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 17:13:15.390177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390147    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 17:13:15.390177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390154    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 17:13:15.390494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390480    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 17:13:15.390551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390486    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 17:13:15.390551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390486    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 17:13:15.390809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390793    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 17:13:15.390879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390830    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 17:13:15.390879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390796    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkzvv\""
Oct 30 17:13:15.390879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390847    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 17:13:15.390879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390837    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 17:13:15.390879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390866    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jw6df\""
Oct 30 17:13:15.390879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390796    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 17:13:15.391115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.390953    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 17:13:15.391115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391010    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 17:13:15.391115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391019    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 17:13:15.391115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391021    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 17:13:15.391115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391098    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 17:13:15.391336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391172    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.391336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391203    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 17:13:15.391336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391243    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rz5ks\""
Oct 30 17:13:15.391336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391322    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.391336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391324    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 17:13:15.391543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391394    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wklnr\""
Oct 30 17:13:15.391543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391411    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 17:13:15.391543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391449    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.391543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391518    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 17:13:15.391543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391531    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 17:13:15.391726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391561    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cr2cr\""
Oct 30 17:13:15.391726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391635    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3n2877180h8a6\""
Oct 30 17:13:15.391726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391720    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zkwh9\""
Oct 30 17:13:15.391821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391758    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.391933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.391916    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.399151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399134    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399135    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399160    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399175    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399186    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399191    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399209    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.399240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399235    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399310    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399369    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399373    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399397    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399399    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399421    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399423    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399451    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399454    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 17:13:15.399556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.399480    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 17:13:15.400127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.400114    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 17:13:15.401120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.401106    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 17:13:15.403007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.402992    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 17:13:15.403622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.403599    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 17:13:15.412398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.412381    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 17:13:15.413138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.413118    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 17:13:15.413294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.413280    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 17:13:15.425380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.425356    5534 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:15.425492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.425463    5534 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:15.425696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.425679    5534 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 17:13:15.448839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.448817    5534 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 17:13:15.448925 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:15.448858    5534 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-170.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:15.449335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.449323    5534 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 17:13:15.449384 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:15.449353    5534 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-170.ec2.internal"
Oct 30 17:13:15.481030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481003    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.481030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481032    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-cni-binary-copy\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.481176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481053    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-node-log\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.481176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481134    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-metrics-client-ca\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.481270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481207    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.481270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481236    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.481270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481263    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-k8s-cni-cncf-io\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.481412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481329    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.481412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481357    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-metrics-client-ca\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.481412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481364    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-cni-binary-copy\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.481556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481505    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.481758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481740    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-netns\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.481824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481794    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-slash\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.481866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481845    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-script-lib\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.481866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481861    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-sys\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.482018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481876    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:15.482018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481869    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.482018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481892    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/376a21c0-10de-43a4-a04c-c3519a868a9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.482018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.481906    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-system-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.482018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482003    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-conf\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482053    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/376a21c0-10de-43a4-a04c-c3519a868a9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482092    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482115    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482132    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482192    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-metrics-certs\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482205    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-kubernetes\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482192    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-script-lib\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482237    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-etc-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482274    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482278    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482299    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482305    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482307    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-config\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482327    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-env-overrides\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482352    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482374    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/412abd52-f4da-491a-b694-30750614e3b4-iptables-alerter-script\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482408    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482434    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-env-overrides\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482463    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-netns\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482486    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-bin\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482518    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovnkube-config\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482514    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d255144b-ecb7-489b-98ca-a188acb6c109-agent-certs\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:13:15.482581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482566    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/412abd52-f4da-491a-b694-30750614e3b4-iptables-alerter-script\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.483233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.482969    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.483233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483012    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa7322d-ff57-4460-be07-755ae180ffe4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.483233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483055    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zh6\" (UniqueName: \"kubernetes.io/projected/caa7322d-ff57-4460-be07-755ae180ffe4-kube-api-access-g4zh6\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.483233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483078    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d255144b-ecb7-489b-98ca-a188acb6c109-agent-certs\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:13:15.483416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483319    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa7322d-ff57-4460-be07-755ae180ffe4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.483416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483324    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.483511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483111    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.483511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483459    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.483511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483496    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.483708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483522    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-multus-certs\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.483708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483554    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-netd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.483708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483585    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa7322d-ff57-4460-be07-755ae180ffe4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.483708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483618    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc79p\" (UniqueName: \"kubernetes.io/projected/34ccfea3-7a41-4925-9669-c4a25efabc4a-kube-api-access-hc79p\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.483708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483666    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-federate-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.483708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483695    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-multus-daemon-config\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.483976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483725    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-lib-modules\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.483976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483778    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovn-node-metrics-cert\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.483976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483808    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltk9\" (UniqueName: \"kubernetes.io/projected/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-kube-api-access-mltk9\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:15.483976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483839    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.483976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.483865    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.484179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484078    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-kubelet-config\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.484179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484109    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6b374b9-fd59-4579-8ebd-fe975116467c-cert\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:13:15.484179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484140    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4hz\" (UniqueName: \"kubernetes.io/projected/a6b374b9-fd59-4579-8ebd-fe975116467c-kube-api-access-9m4hz\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:13:15.484179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484173    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.484346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484206    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-systemd-units\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.484346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484233    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/908859a0-1e77-4a62-a253-3486242475d8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.484346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484262    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.484346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484294    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.484346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484326    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484349    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484379    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-kubelet\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484408    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-var-lib-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484436    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6a516e5-f016-4d54-be70-b1979fe0c605-metrics-tls\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484466    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-etc-kubernetes\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484492    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c926e-5a6b-4c44-881c-5dc47879a022-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484522    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-systemd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.484556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484551    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-etc-tuned\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484583    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddxd\" (UniqueName: \"kubernetes.io/projected/caa54cca-be01-4afd-8690-5cb0958c1027-kube-api-access-5ddxd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484620    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484661    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-registration-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484700    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484724    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484756    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-system-cni-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484787    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-hostroot\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484816    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-bin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484840    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-conf-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484868    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.484907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484899    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484927    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/412abd52-f4da-491a-b694-30750614e3b4-host-slash\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484956    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-serving-certs-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.484982    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485018    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-cnibin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485048    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485076    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485104    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-dbus\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485127    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-modprobe-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485155    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysconfig\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485180    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-var-lib-kubelet\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485211    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485238    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxqx\" (UniqueName: \"kubernetes.io/projected/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-kube-api-access-srxqx\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485267    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485296    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-socket-dir-parent\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485326    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-textfile\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.485364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485356    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwlt\" (UniqueName: \"kubernetes.io/projected/7a3c7100-3ff5-4410-abd7-03e68c27baa3-kube-api-access-sxwlt\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485377    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-sys\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485408    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa7322d-ff57-4460-be07-755ae180ffe4-crio-socket\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485435    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-host\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485468    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485496    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-device-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485522    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zv5\" (UniqueName: \"kubernetes.io/projected/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kube-api-access-j6zv5\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485552    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-wtmp\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485582    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485611    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlt82\" (UniqueName: \"kubernetes.io/projected/322b7165-7398-4ed9-99a9-cef5e8ee3c86-kube-api-access-wlt82\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485657    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-tmp\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485695    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-log-socket\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485726    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485756    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a3c7100-3ff5-4410-abd7-03e68c27baa3-serviceca\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485782    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lhx\" (UniqueName: \"kubernetes.io/projected/412abd52-f4da-491a-b694-30750614e3b4-kube-api-access-x5lhx\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485812    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-root\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485844    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485877    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485909    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-socket-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485935    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-tls\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.485967    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8wh\" (UniqueName: \"kubernetes.io/projected/376a21c0-10de-43a4-a04c-c3519a868a9b-kube-api-access-rb8wh\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486000    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhq7\" (UniqueName: \"kubernetes.io/projected/908859a0-1e77-4a62-a253-3486242475d8-kube-api-access-6vhq7\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486033    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d255144b-ecb7-489b-98ca-a188acb6c109-konnectivity-ca\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486065    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-sys-fs\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.486085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486093    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-metrics-client-ca\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486123    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-cnibin\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486153    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486184    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcdl\" (UniqueName: \"kubernetes.io/projected/84532c65-2d04-45a5-b0cb-9841d8e20d87-kube-api-access-mlcdl\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486213    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486246    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486276    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-multus\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486306    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccb7x\" (UniqueName: \"kubernetes.io/projected/df1c926e-5a6b-4c44-881c-5dc47879a022-kube-api-access-ccb7x\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486338    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a516e5-f016-4d54-be70-b1979fe0c605-config-volume\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486367    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpshs\" (UniqueName: \"kubernetes.io/projected/991f66af-6bcb-4bd8-921f-58f2fb879067-kube-api-access-fpshs\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486395    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a3c7100-3ff5-4410-abd7-03e68c27baa3-host\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486426    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486457    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzl9\" (UniqueName: \"kubernetes.io/projected/622e2edd-70ee-4176-8498-50a740a59f82-kube-api-access-4mzl9\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486487    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-run\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486511    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/991f66af-6bcb-4bd8-921f-58f2fb879067-hosts-file\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486548    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486590    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486623    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxr4\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-kube-api-access-8gxr4\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486671    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486702    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-systemd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486730    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-ovn\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486759    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a516e5-f016-4d54-be70-b1979fe0c605-tmp-dir\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.487091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.486976    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487115    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487311    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c926e-5a6b-4c44-881c-5dc47879a022-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487335    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84532c65-2d04-45a5-b0cb-9841d8e20d87-ovn-node-metrics-cert\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487348    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487354    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487381    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487396    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a516e5-f016-4d54-be70-b1979fe0c605-tmp-dir\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487413    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-etc-tuned\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487527    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa54cca-be01-4afd-8690-5cb0958c1027-tmp\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487610    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa7322d-ff57-4460-be07-755ae180ffe4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487684    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487703    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-tls\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487797    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9k9\" (UniqueName: \"kubernetes.io/projected/a6a516e5-f016-4d54-be70-b1979fe0c605-kube-api-access-tl9k9\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.488088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.487841    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa7322d-ff57-4460-be07-755ae180ffe4-data-volume\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488171    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-kubelet\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488189    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/908859a0-1e77-4a62-a253-3486242475d8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488206    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488219    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f66af-6bcb-4bd8-921f-58f2fb879067-tmp-dir\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488258    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488275    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-serving-certs-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488283    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d255144b-ecb7-489b-98ca-a188acb6c109-konnectivity-ca\") pod \"konnectivity-agent-4hzkr\" (UID: \"d255144b-ecb7-489b-98ca-a188acb6c109\") " pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488411    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488437    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/622e2edd-70ee-4176-8498-50a740a59f82-multus-daemon-config\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488549    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a516e5-f016-4d54-be70-b1979fe0c605-config-volume\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488563    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.488762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488597    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.489252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488788    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.489252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488854    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-federate-client-tls\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.489252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488899    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-textfile\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.489398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489352    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-original-pull-secret\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.489453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489420    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.489453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489424    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.489453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489444    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-metrics-client-ca\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.489601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489472    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.489601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489492    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.489601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489543    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.489601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489544    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34ccfea3-7a41-4925-9669-c4a25efabc4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.489601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489581    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.489601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489598    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.489903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489614    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.489903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489691    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6a516e5-f016-4d54-be70-b1979fe0c605-metrics-tls\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.489903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489696    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f66af-6bcb-4bd8-921f-58f2fb879067-tmp-dir\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.489903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489722    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6b374b9-fd59-4579-8ebd-fe975116467c-cert\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:13:15.489903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489826    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.489903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.489838    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a3c7100-3ff5-4410-abd7-03e68c27baa3-serviceca\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.490174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488549    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.490174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.488741    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.490174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.490144    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/908859a0-1e77-4a62-a253-3486242475d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.490174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.490145    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/376a21c0-10de-43a4-a04c-c3519a868a9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.490362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.490183    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa7322d-ff57-4460-be07-755ae180ffe4-data-volume\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.490362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.490196    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.490950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.490933    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.504469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.504453    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zh6\" (UniqueName: \"kubernetes.io/projected/caa7322d-ff57-4460-be07-755ae180ffe4-kube-api-access-g4zh6\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.507762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.507742    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlt82\" (UniqueName: \"kubernetes.io/projected/322b7165-7398-4ed9-99a9-cef5e8ee3c86-kube-api-access-wlt82\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.507891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.507878    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxqx\" (UniqueName: \"kubernetes.io/projected/4c86d1f6-23f6-4d70-bcac-400d7c08bc09-kube-api-access-srxqx\") pod \"telemeter-client-85f86dfd46-lsb6t\" (UID: \"4c86d1f6-23f6-4d70-bcac-400d7c08bc09\") " pod="openshift-monitoring/telemeter-client-85f86dfd46-lsb6t"
Oct 30 17:13:15.510519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.510504    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zv5\" (UniqueName: \"kubernetes.io/projected/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kube-api-access-j6zv5\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.510637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.510621    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddxd\" (UniqueName: \"kubernetes.io/projected/caa54cca-be01-4afd-8690-5cb0958c1027-kube-api-access-5ddxd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.513897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.513876    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccb7x\" (UniqueName: \"kubernetes.io/projected/df1c926e-5a6b-4c44-881c-5dc47879a022-kube-api-access-ccb7x\") pod \"monitoring-plugin-77f4f76594-mxs5v\" (UID: \"df1c926e-5a6b-4c44-881c-5dc47879a022\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:15.514604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514157    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9k9\" (UniqueName: \"kubernetes.io/projected/a6a516e5-f016-4d54-be70-b1979fe0c605-kube-api-access-tl9k9\") pod \"dns-default-rcllv\" (UID: \"a6a516e5-f016-4d54-be70-b1979fe0c605\") " pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:15.514604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514454    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4hz\" (UniqueName: \"kubernetes.io/projected/a6b374b9-fd59-4579-8ebd-fe975116467c-kube-api-access-9m4hz\") pod \"ingress-canary-pwswg\" (UID: \"a6b374b9-fd59-4579-8ebd-fe975116467c\") " pod="openshift-ingress-canary/ingress-canary-pwswg"
Oct 30 17:13:15.514604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514575    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpshs\" (UniqueName: \"kubernetes.io/projected/991f66af-6bcb-4bd8-921f-58f2fb879067-kube-api-access-fpshs\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.514604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514582    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzl9\" (UniqueName: \"kubernetes.io/projected/622e2edd-70ee-4176-8498-50a740a59f82-kube-api-access-4mzl9\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.514800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514583    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lhx\" (UniqueName: \"kubernetes.io/projected/412abd52-f4da-491a-b694-30750614e3b4-kube-api-access-x5lhx\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.514800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514623    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwlt\" (UniqueName: \"kubernetes.io/projected/7a3c7100-3ff5-4410-abd7-03e68c27baa3-kube-api-access-sxwlt\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.514800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514661    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8wh\" (UniqueName: \"kubernetes.io/projected/376a21c0-10de-43a4-a04c-c3519a868a9b-kube-api-access-rb8wh\") pod \"openshift-state-metrics-7bd7489957-kdm8l\" (UID: \"376a21c0-10de-43a4-a04c-c3519a868a9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kdm8l"
Oct 30 17:13:15.514800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514669    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drb5c\" (UniqueName: \"kubernetes.io/projected/cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a-kube-api-access-drb5c\") pod \"network-check-target-z2hmf\" (UID: \"cad1bce8-8076-4ed0-a90d-73e5fd9b4b4a\") " pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:15.514800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514695    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxr4\" (UniqueName: \"kubernetes.io/projected/9dca0b75-8ca5-4e65-8c40-75ca747fffb3-kube-api-access-8gxr4\") pod \"prometheus-k8s-0\" (UID: \"9dca0b75-8ca5-4e65-8c40-75ca747fffb3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:15.514940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.514872    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcdl\" (UniqueName: \"kubernetes.io/projected/84532c65-2d04-45a5-b0cb-9841d8e20d87-kube-api-access-mlcdl\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.515677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.515661    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhq7\" (UniqueName: \"kubernetes.io/projected/908859a0-1e77-4a62-a253-3486242475d8-kube-api-access-6vhq7\") pod \"kube-state-metrics-5dc8d7d89b-qczgg\" (UID: \"908859a0-1e77-4a62-a253-3486242475d8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qczgg"
Oct 30 17:13:15.516226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.516208    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltk9\" (UniqueName: \"kubernetes.io/projected/a5a1e416-7f8a-4940-85c2-19f0fd39c5f4-kube-api-access-mltk9\") pod \"network-metrics-daemon-dqkvr\" (UID: \"a5a1e416-7f8a-4940-85c2-19f0fd39c5f4\") " pod="openshift-multus/network-metrics-daemon-dqkvr"
Oct 30 17:13:15.519481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.519465    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc79p\" (UniqueName: \"kubernetes.io/projected/34ccfea3-7a41-4925-9669-c4a25efabc4a-kube-api-access-hc79p\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.589457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589388    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-system-cni-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.589457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589430    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-hostroot\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589453    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-bin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589480    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-conf-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589501    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589502    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-system-cni-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589525    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/412abd52-f4da-491a-b694-30750614e3b4-host-slash\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589537    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-hostroot\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589546    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-cnibin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589571    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-conf-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589586    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-dbus\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589599    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-cnibin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589608    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-modprobe-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589616    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-bin\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589636    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysconfig\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589680    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-var-lib-kubelet\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589693    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-dbus\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.589698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589706    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589727    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-modprobe-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589732    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-socket-dir-parent\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589761    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589799    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-sys\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589807    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589823    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-socket-dir-parent\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysconfig\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589855    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/412abd52-f4da-491a-b694-30750614e3b4-host-slash\") pod \"iptables-alerter-6nccg\" (UID: \"412abd52-f4da-491a-b694-30750614e3b4\") " pod="openshift-network-operator/iptables-alerter-6nccg"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589827    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa7322d-ff57-4460-be07-755ae180ffe4-crio-socket\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589869    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa7322d-ff57-4460-be07-755ae180ffe4-crio-socket\") pod \"insights-runtime-extractor-p274z\" (UID: \"caa7322d-ff57-4460-be07-755ae180ffe4\") " pod="openshift-insights/insights-runtime-extractor-p274z"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589867    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-sys\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589920    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-host\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589808    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-var-lib-kubelet\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589990    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-host\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.589995    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-device-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590051    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-wtmp\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590080    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-log-socket\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590084    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-device-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590096    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-root\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590115    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590138    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-socket-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590165    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-sys-fs\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590189    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-cnibin\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590216    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-multus\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590243    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a3c7100-3ff5-4410-abd7-03e68c27baa3-host\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590243    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-node-exporter-wtmp\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590264    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-run\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590283    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-log-socket\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-sys-fs\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590287    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/991f66af-6bcb-4bd8-921f-58f2fb879067-hosts-file\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590328    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-cni-multus\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590328    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590357    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a3c7100-3ff5-4410-abd7-03e68c27baa3-host\") pod \"node-ca-xjh5h\" (UID: \"7a3c7100-3ff5-4410-abd7-03e68c27baa3\") " pod="openshift-image-registry/node-ca-xjh5h"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590371    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/991f66af-6bcb-4bd8-921f-58f2fb879067-hosts-file\") pod \"node-resolver-zz6f9\" (UID: \"991f66af-6bcb-4bd8-921f-58f2fb879067\") " pod="openshift-dns/node-resolver-zz6f9"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590376    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-run\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590394    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34ccfea3-7a41-4925-9669-c4a25efabc4a-cnibin\") pod \"multus-additional-cni-plugins-5xbwb\" (UID: \"34ccfea3-7a41-4925-9669-c4a25efabc4a\") " pod="openshift-multus/multus-additional-cni-plugins-5xbwb"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590405    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-root\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590421    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-systemd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590448    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-socket-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.590455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590487    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-ovn\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590509    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-systemd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590573    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-ovn\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590593    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590634    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-multus-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590659    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-kubelet\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590687    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-var-lib-kubelet\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590690    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-node-log\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590723    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-node-log\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590725    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-k8s-cni-cncf-io\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590748    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-k8s-cni-cncf-io\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590759    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-netns\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590783    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-slash\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590798    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-netns\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590811    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-sys\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590834    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-slash\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590838    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-system-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590857    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/322b7165-7398-4ed9-99a9-cef5e8ee3c86-sys\") pod \"node-exporter-mbfgv\" (UID: \"322b7165-7398-4ed9-99a9-cef5e8ee3c86\") " pod="openshift-monitoring/node-exporter-mbfgv"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590860    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-conf\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590885    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-kubernetes\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590897    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-system-cni-dir\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590908    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-etc-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590933    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590959    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-etc-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590934    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-kubernetes\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590987    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-ovn-kubernetes\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590960    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591002    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591021    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-netns\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591038    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-bin\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.590991    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-conf\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591061    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-multus-certs\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591075    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-run-netns\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591089    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-netd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591093    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-bin\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591101    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-host-run-multus-certs\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591116    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-lib-modules\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591134    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-cni-netd\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591144    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-kubelet-config\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591171    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-systemd-units\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591183    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3dcc262-c93f-43f7-ba5f-601ac5ab06b3-kubelet-config\") pod \"global-pull-secret-syncer-v8rgj\" (UID: \"f3dcc262-c93f-43f7-ba5f-601ac5ab06b3\") " pod="kube-system/global-pull-secret-syncer-v8rgj"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-lib-modules\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591201    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591223    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-systemd-units\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591225    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-kubelet\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591242    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-host-kubelet\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591257    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-var-lib-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591280    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-etc-kubernetes\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591298    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-sysctl-d\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591306    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-systemd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591337    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591343    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622e2edd-70ee-4176-8498-50a740a59f82-etc-kubernetes\") pod \"multus-7k5d2\" (UID: \"622e2edd-70ee-4176-8498-50a740a59f82\") " pod="openshift-multus/multus-7k5d2"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591362    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-registration-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591379    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-run-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591404    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caa54cca-be01-4afd-8690-5cb0958c1027-etc-systemd\") pod \"tuned-nc4z8\" (UID: \"caa54cca-be01-4afd-8690-5cb0958c1027\") " pod="openshift-cluster-node-tuning-operator/tuned-nc4z8"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591363    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84532c65-2d04-45a5-b0cb-9841d8e20d87-var-lib-openvswitch\") pod \"ovnkube-node-96dfd\" (UID: \"84532c65-2d04-45a5-b0cb-9841d8e20d87\") " pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:15.591918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:15.591414    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a1c6d38-094f-41a0-8c77-b1f54d231ea7-registration-dir\") pod \"aws-ebs-csi-driver-node-nwllw\" (UID: \"6a1c6d38-094f-41a0-8c77-b1f54d231ea7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwllw"
Oct 30 17:13:16.489769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:16.489739    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:16.508674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:16.508634    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:17.689412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:17.689384    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rcllv"
Oct 30 17:13:18.178576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:18.178546    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:19.617669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:19.617617    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:19.618049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:19.617836    5534 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 17:13:21.043000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:21.042973    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxs5v"
Oct 30 17:13:21.165893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:21.165864    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:21.766501 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:21.766457    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:21.877472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:21.877445    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:22.725463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:22.725432    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hzkr"
Oct 30 17:13:25.303624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:25.303528    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:13:26.769473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:26.769421    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:26.878171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:26.878140    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:27.209025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:27.208992    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:35.949024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:35.948976    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:36.059428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:36.059378    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:36.262902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:36.262780    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:36.371229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:36.371162    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:36.671467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:36.671408    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:36.797878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:36.797838    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:38.864879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:38.864845    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-96dfd"
Oct 30 17:13:44.618862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:44.618819    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:44.765588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:44.765536    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:45.241617 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:13:45.241585    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:13:51.169072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:51.169042    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z2hmf"
Oct 30 17:13:51.674918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:51.674880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:13:51.783706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:13:51.783673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:14:12.238987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:14:12.238952    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:14:15.242626 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:14:15.242582    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:14:16.411183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:14:16.411145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:14:20.989733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:14:20.989700    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:14:25.319630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:14:25.319600    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 17:14:45.014899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:14:45.014861    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:14:45.125717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:14:45.125683    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:14:45.242021 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:14:45.241983    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:15:12.368246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:15:12.368168    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:15:15.242996 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:15:15.242959    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:15:23.421093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:15:23.421062    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:15:45.242399 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:15:45.242357    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:16:03.489575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:03.489537    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:03.621830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:03.621796    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:12.458556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:12.458522    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:15.243003 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:16:15.242965    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:16:16.355909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:16.355875    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:16.485572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:16.485535    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:35.599868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:35.599788    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:37.022828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:37.022787    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:38.806487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:38.806450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:38.939408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:38.939359    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:40.855969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:40.855927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:40.999577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:40.999548    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:41.173133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:41.173021    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:43.272993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:43.272961    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:44.558251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:44.558223    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:45.243277 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:16:45.243239    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:16:45.998261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:45.998195    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:46.109590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:46.109559    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:48.493222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:48.493195    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:48.605614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:48.605585    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:51.271067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.271040    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:51.552860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.552781    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-ntp4k"]
Oct 30 17:16:51.554971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.554950    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.582315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.582281    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 17:16:51.582315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.582304    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 17:16:51.582315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.582311    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bwt85\""
Oct 30 17:16:51.582846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.582828    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ntp4k"]
Oct 30 17:16:51.619821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.619778    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4rs\" (UniqueName: \"kubernetes.io/projected/8b93df17-97e8-4552-ac26-3a697c92b3db-kube-api-access-7b4rs\") pod \"cert-manager-597b95cb84-ntp4k\" (UID: \"8b93df17-97e8-4552-ac26-3a697c92b3db\") " pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.620016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.619860    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b93df17-97e8-4552-ac26-3a697c92b3db-bound-sa-token\") pod \"cert-manager-597b95cb84-ntp4k\" (UID: \"8b93df17-97e8-4552-ac26-3a697c92b3db\") " pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.720667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.720599    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4rs\" (UniqueName: \"kubernetes.io/projected/8b93df17-97e8-4552-ac26-3a697c92b3db-kube-api-access-7b4rs\") pod \"cert-manager-597b95cb84-ntp4k\" (UID: \"8b93df17-97e8-4552-ac26-3a697c92b3db\") " pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.720831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.720695    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b93df17-97e8-4552-ac26-3a697c92b3db-bound-sa-token\") pod \"cert-manager-597b95cb84-ntp4k\" (UID: \"8b93df17-97e8-4552-ac26-3a697c92b3db\") " pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.741552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.741516    5534 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 17:16:51.744145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.744102    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b93df17-97e8-4552-ac26-3a697c92b3db-bound-sa-token\") pod \"cert-manager-597b95cb84-ntp4k\" (UID: \"8b93df17-97e8-4552-ac26-3a697c92b3db\") " pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.744276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.744219    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4rs\" (UniqueName: \"kubernetes.io/projected/8b93df17-97e8-4552-ac26-3a697c92b3db-kube-api-access-7b4rs\") pod \"cert-manager-597b95cb84-ntp4k\" (UID: \"8b93df17-97e8-4552-ac26-3a697c92b3db\") " pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:51.876216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:51.876185    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ntp4k"
Oct 30 17:16:52.169243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:52.169178    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ntp4k"]
Oct 30 17:16:52.172289 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:16:52.172258    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b93df17_97e8_4552_ac26_3a697c92b3db.slice/crio-69969801f62987075e5e4f7b488de5d19bb26d2ce037c2bf796324a7cda79a50 WatchSource:0}: Error finding container 69969801f62987075e5e4f7b488de5d19bb26d2ce037c2bf796324a7cda79a50: Status 404 returned error can't find the container with id 69969801f62987075e5e4f7b488de5d19bb26d2ce037c2bf796324a7cda79a50
Oct 30 17:16:52.174151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:52.174133    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:16:53.136700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:53.136663    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ntp4k" event={"ID":"8b93df17-97e8-4552-ac26-3a697c92b3db","Type":"ContainerStarted","Data":"69969801f62987075e5e4f7b488de5d19bb26d2ce037c2bf796324a7cda79a50"}
Oct 30 17:16:55.145519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:55.145485    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ntp4k" event={"ID":"8b93df17-97e8-4552-ac26-3a697c92b3db","Type":"ContainerStarted","Data":"5209019e2d862bd269ce817009d97fd73f33d68526fec7d6e265992e6966cb8e"}
Oct 30 17:16:55.184855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:55.184807    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-ntp4k" podStartSLOduration=1.4189517839999999 podStartE2EDuration="4.184794402s" totalImagesPullingTime="2.765842618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:55.184074261 +0000 UTC m=+220.946118075" watchObservedRunningTime="2026-10-30 17:16:55.184794402 +0000 UTC m=+220.946838215"
Oct 30 17:16:57.801940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:57.801901    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:16:57.911918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:16:57.911881    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:03.477957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:03.477803    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:03.609104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:03.609067    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:05.257541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:05.257510    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:10.381745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:10.381715    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:10.490393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:10.490367    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:12.582907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:12.582873    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:13.333755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:13.333716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:13.508290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:13.508262    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:15.242012 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:17:15.241973    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:17:16.852183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:16.852149    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:16.967476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:16.967448    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:17.432589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.432553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:17.587492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.587433    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:17.649466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.649433    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"]
Oct 30 17:17:17.651633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.651614    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.690596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.690537    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"]
Oct 30 17:17:17.739615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.739586    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 17:17:17.739762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.739626    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 17:17:17.744829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.744810    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 17:17:17.744919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.744839    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 17:17:17.744919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.744858    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 17:17:17.744919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.744881    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qkrcd\""
Oct 30 17:17:17.787416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.787392    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:17.832396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.832370    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.832489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.832472    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2xt\" (UniqueName: \"kubernetes.io/projected/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-kube-api-access-bw2xt\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.832531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.832518    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.832572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.832559    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.832680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.832664    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.832730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.832694    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.901830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.901799    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:17.932943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.932919    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.933050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.932952    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2xt\" (UniqueName: \"kubernetes.io/projected/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-kube-api-access-bw2xt\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.933050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.932978    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.933137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.933094    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.933181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.933152    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.933218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.933187    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.933515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.933483    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.935451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.935433    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.935571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.935554    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.935690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.935670    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.935803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.935784    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.955929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.955878    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2xt\" (UniqueName: \"kubernetes.io/projected/8a175b30-bdc4-42a1-b81b-0f5d9da955c0-kube-api-access-bw2xt\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg\" (UID: \"8a175b30-bdc4-42a1-b81b-0f5d9da955c0\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:17.963730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:17.963715    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:18.018386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:18.018355    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:18.182389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:18.182361    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:18.318458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:18.318380    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"]
Oct 30 17:17:18.321423 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:17:18.321392    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a175b30_bdc4_42a1_b81b_0f5d9da955c0.slice/crio-9ddb24e054cde42b38d49569124d1d49f7975ad356343e05e61ead8a33785289 WatchSource:0}: Error finding container 9ddb24e054cde42b38d49569124d1d49f7975ad356343e05e61ead8a33785289: Status 404 returned error can't find the container with id 9ddb24e054cde42b38d49569124d1d49f7975ad356343e05e61ead8a33785289
Oct 30 17:17:19.225976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:19.225934    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg" event={"ID":"8a175b30-bdc4-42a1-b81b-0f5d9da955c0","Type":"ContainerStarted","Data":"9ddb24e054cde42b38d49569124d1d49f7975ad356343e05e61ead8a33785289"}
Oct 30 17:17:20.007171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:20.007136    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:20.130716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:20.130676    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:22.239164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:22.239128    5534 generic.go:356] "Generic (PLEG): container finished" podID="8a175b30-bdc4-42a1-b81b-0f5d9da955c0" containerID="ca64063da4067497af68b309c335379fdee9cad8d40583fbc5b9484548bbbda6" exitCode=0
Oct 30 17:17:22.239164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:22.239166    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg" event={"ID":"8a175b30-bdc4-42a1-b81b-0f5d9da955c0","Type":"ContainerDied","Data":"ca64063da4067497af68b309c335379fdee9cad8d40583fbc5b9484548bbbda6"}
Oct 30 17:17:23.245800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:23.245759    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg" event={"ID":"8a175b30-bdc4-42a1-b81b-0f5d9da955c0","Type":"ContainerStarted","Data":"fbc8bcbc7c0a483f502480f420a6575796f37e37493db834f208c1234d9bab33"}
Oct 30 17:17:23.246254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:23.245911    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:23.349145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:23.349101    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg" podStartSLOduration=2.649144052 podStartE2EDuration="6.349085853s" totalImagesPullingTime="3.207397584s" totalInitContainerRuntime="492.544217ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:23.336442347 +0000 UTC m=+249.098486159" watchObservedRunningTime="2026-10-30 17:17:23.349085853 +0000 UTC m=+249.111129665"
Oct 30 17:17:31.793093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:31.793059    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:31.913722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:31.913692    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:32.028142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:32.028114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:34.250837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:34.250809    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg"
Oct 30 17:17:38.244431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.244387    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"]
Oct 30 17:17:38.253597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.253574    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.267628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.267604    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"]
Oct 30 17:17:38.278749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.278726    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 17:17:38.278749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.278741    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 17:17:38.281842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.281818    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 17:17:38.281943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.281842    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 17:17:38.281943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.281891    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xwfmd\""
Oct 30 17:17:38.281943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.281932    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 17:17:38.285206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.285188    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-cert\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.285323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.285235    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwj6\" (UniqueName: \"kubernetes.io/projected/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-kube-api-access-qqwj6\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.285323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.285255    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-metrics-cert\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.285323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.285311    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-manager-config\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.386653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.386615    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwj6\" (UniqueName: \"kubernetes.io/projected/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-kube-api-access-qqwj6\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.386803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.386684    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-metrics-cert\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.386803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.386721    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-manager-config\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.386934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.386917    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-cert\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.387491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.387467    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-manager-config\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.389372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.389344    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-metrics-cert\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.389469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.389372    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-cert\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.409618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.409592    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwj6\" (UniqueName: \"kubernetes.io/projected/e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3-kube-api-access-qqwj6\") pod \"lws-controller-manager-6c446c6ff5-fqbgt\" (UID: \"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.564719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.564610    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:38.860315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:38.860271    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"]
Oct 30 17:17:38.863353 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:17:38.863326    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c99a79_5e8e_4ac2_86d6_3f40179f8fd3.slice/crio-1ad1e57086e6393654eb3df261a2148201482c0f3a7df92d0047c9bad8e79aef WatchSource:0}: Error finding container 1ad1e57086e6393654eb3df261a2148201482c0f3a7df92d0047c9bad8e79aef: Status 404 returned error can't find the container with id 1ad1e57086e6393654eb3df261a2148201482c0f3a7df92d0047c9bad8e79aef
Oct 30 17:17:39.305910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:39.305823    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt" event={"ID":"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3","Type":"ContainerStarted","Data":"1ad1e57086e6393654eb3df261a2148201482c0f3a7df92d0047c9bad8e79aef"}
Oct 30 17:17:42.318587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:42.318551    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt" event={"ID":"e0c99a79-5e8e-4ac2-86d6-3f40179f8fd3","Type":"ContainerStarted","Data":"bef3994cce7aeaa0c7100d2e51c55540860f4612240705f02cd92f0b946bbd67"}
Oct 30 17:17:42.319026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:42.318692    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:42.358792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:42.358735    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt" podStartSLOduration=1.895937536 podStartE2EDuration="4.358718897s" totalImagesPullingTime="2.462781361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:42.358337485 +0000 UTC m=+268.120381298" watchObservedRunningTime="2026-10-30 17:17:42.358718897 +0000 UTC m=+268.120762711"
Oct 30 17:17:45.241947 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:17:45.241909    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:17:46.983706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:46.983666    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:47.125664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.125321    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:47.526129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.526083    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:47.537263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.537234    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"]
Oct 30 17:17:47.541747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.541726    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.563129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.563106    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"]
Oct 30 17:17:47.573817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.573787    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 17:17:47.573817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.573800    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 17:17:47.574135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.574090    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j56w8\""
Oct 30 17:17:47.574135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.574093    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 17:17:47.654203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.654165    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:47.662004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.661980    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5v7c\" (UniqueName: \"kubernetes.io/projected/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-kube-api-access-l5v7c\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.662101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.662007    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 17:17:47.662101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.662012    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-tls-certs\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.662101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.662072    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-tmp\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.762929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.762892    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5v7c\" (UniqueName: \"kubernetes.io/projected/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-kube-api-access-l5v7c\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.763101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.762953    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-tls-certs\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.763101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.763002    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-tmp\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.765413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.765386    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-tmp\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.765593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.765574    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-tls-certs\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.785880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.785830    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5v7c\" (UniqueName: \"kubernetes.io/projected/e8a85129-d6fd-42c0-b2a3-50187e0edf0a-kube-api-access-l5v7c\") pod \"kube-auth-proxy-57db9d4f85-fc579\" (UID: \"e8a85129-d6fd-42c0-b2a3-50187e0edf0a\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:47.854158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:47.854104    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"
Oct 30 17:17:48.149285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:48.149262    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57db9d4f85-fc579"]
Oct 30 17:17:48.151793 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:17:48.151767    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a85129_d6fd_42c0_b2a3_50187e0edf0a.slice/crio-cfd3c97f0d1dd9bafe596fa36def5c1ca6fa52e622fd09e4cd2de9cfa8e755a7 WatchSource:0}: Error finding container cfd3c97f0d1dd9bafe596fa36def5c1ca6fa52e622fd09e4cd2de9cfa8e755a7: Status 404 returned error can't find the container with id cfd3c97f0d1dd9bafe596fa36def5c1ca6fa52e622fd09e4cd2de9cfa8e755a7
Oct 30 17:17:48.340384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:48.340352    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579" event={"ID":"e8a85129-d6fd-42c0-b2a3-50187e0edf0a","Type":"ContainerStarted","Data":"cfd3c97f0d1dd9bafe596fa36def5c1ca6fa52e622fd09e4cd2de9cfa8e755a7"}
Oct 30 17:17:49.171347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:49.171312    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:49.289270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:49.289203    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:50.543686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:50.543629    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:50.917504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:50.917474    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:51.236679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:51.236596    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:51.361032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:51.360998    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579" event={"ID":"e8a85129-d6fd-42c0-b2a3-50187e0edf0a","Type":"ContainerStarted","Data":"5b0e8edd58d6df2741c17f5f44057db9da5074e3d6609ba1706de96241eb833d"}
Oct 30 17:17:51.406379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:51.406313    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-fc579" podStartSLOduration=1.990139678 podStartE2EDuration="4.406294084s" totalImagesPullingTime="2.416154406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:51.406162569 +0000 UTC m=+277.168206384" watchObservedRunningTime="2026-10-30 17:17:51.406294084 +0000 UTC m=+277.168337901"
Oct 30 17:17:51.603263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:51.603227    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:53.140786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:53.140746    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:53.260863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:53.260829    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:53.325504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:53.325479    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-fqbgt"
Oct 30 17:17:54.157226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:54.157190    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:54.296682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:54.296631    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:17:54.588242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:17:54.588144    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:18:12.756484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:18:12.756451    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:18:15.243493 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:18:15.243455    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:18:15.495707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:18:15.495606    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:18:15.726464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:18:15.726436    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:18:45.246255 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:18:45.246222    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:19:02.682207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:02.682171    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:02.807064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:02.807029    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:02.951706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:02.951613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:03.087419 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:03.087385    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:03.403317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:03.403281    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:05.242215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:05.242174    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:05.359435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:05.359405    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:05.468462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:05.468430    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:05.580963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:05.580881    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:12.868789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:12.868754    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:15.243455 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:19:15.243420    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:19:17.287614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:17.287579    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:17.407997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:17.407960    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:17.711102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:17.711066    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:45.243510 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:19:45.243424    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:19:50.543744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:50.543707    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:19:50.673272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:19:50.673239    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:01.472287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:01.472251    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:02.263797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:02.263765    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:05.481498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:05.481460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:05.719897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:05.719865    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:05.848120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:05.848084    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:07.215224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:07.215189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:07.656589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:07.656550    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:08.063881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:08.063778    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:08.504590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:08.504551    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:08.785082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:08.784984    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:11.865523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:11.865484    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:13.016962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:13.016927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:13.365073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:13.365046    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:14.870552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:14.870510    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:15.010011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:15.009974    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:15.247243 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:15.247138    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:20:15.275905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:15.275854    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:15.450224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:15.450187    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:15.673525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:15.673490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:16.650921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:16.650885    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:17.824605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:17.824569    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:18.670799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.670768    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:18.782905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.782870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:18.833324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.833292    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"]
Oct 30 17:20:18.836316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.836296    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:18.852809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.852781    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 17:20:18.855510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.855489    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2wkrd\""
Oct 30 17:20:18.864979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.864942    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 17:20:18.864979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.864947    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 17:20:18.867532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.867504    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 17:20:18.868375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.868355    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"]
Oct 30 17:20:18.901336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.901301    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:18.935786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.935701    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:18.935786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.935760    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ntrm\" (UniqueName: \"kubernetes.io/projected/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-kube-api-access-8ntrm\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:18.935969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:18.935851    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.018793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.018763    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:19.036898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.036864    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.037100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.036907    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.037100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.036954    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ntrm\" (UniqueName: \"kubernetes.io/projected/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-kube-api-access-8ntrm\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.037519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.037498    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.039486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.039466    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.060928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.060892    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ntrm\" (UniqueName: \"kubernetes.io/projected/7ba1c859-3d6d-47d5-8ab1-72cba22b2146-kube-api-access-8ntrm\") pod \"kuadrant-console-plugin-66c4fb575d-26cq7\" (UID: \"7ba1c859-3d6d-47d5-8ab1-72cba22b2146\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.152935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.152889    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"
Oct 30 17:20:19.455118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.455047    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7"]
Oct 30 17:20:19.458284 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:20:19.458257    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ba1c859_3d6d_47d5_8ab1_72cba22b2146.slice/crio-84dbd81611b0eff209c6fbe5416353e369074435453eed66691f2b154a75bd3c WatchSource:0}: Error finding container 84dbd81611b0eff209c6fbe5416353e369074435453eed66691f2b154a75bd3c: Status 404 returned error can't find the container with id 84dbd81611b0eff209c6fbe5416353e369074435453eed66691f2b154a75bd3c
Oct 30 17:20:19.871520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:19.871484    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7" event={"ID":"7ba1c859-3d6d-47d5-8ab1-72cba22b2146","Type":"ContainerStarted","Data":"84dbd81611b0eff209c6fbe5416353e369074435453eed66691f2b154a75bd3c"}
Oct 30 17:20:24.181447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:24.181360    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:24.293800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:24.293759    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:24.860753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:24.860703    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:24.891182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:24.891147    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7" event={"ID":"7ba1c859-3d6d-47d5-8ab1-72cba22b2146","Type":"ContainerStarted","Data":"7264954ab91a1446e4869bdc608bfb0db1a21f05c07d2879f67cefc76b9ea0be"}
Oct 30 17:20:24.929955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:24.929895    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-26cq7" podStartSLOduration=2.528728063 podStartE2EDuration="6.929882405s" totalImagesPullingTime="4.401154342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:24.929322267 +0000 UTC m=+430.691366082" watchObservedRunningTime="2026-10-30 17:20:24.929882405 +0000 UTC m=+430.691926233"
Oct 30 17:20:24.970538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:24.970511    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:25.462073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:25.462039    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:25.661182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:25.661150    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:28.974406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:28.974368    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"]
Oct 30 17:20:28.977582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:28.977565    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.008022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.007995    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4ptvx\""
Oct 30 17:20:29.008798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.008778    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"]
Oct 30 17:20:29.033732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.033687    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrc7d\" (UniqueName: \"kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d\") pod \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.033904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.033746    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.134931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.134893    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrc7d\" (UniqueName: \"kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d\") pod \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.135193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.135173    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.135772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.135748    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.160933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.160906    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrc7d\" (UniqueName: \"kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d\") pod \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.290146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.290060    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:29.625292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.625264    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"]
Oct 30 17:20:29.627981 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:20:29.627950    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589a8b96_db1d_4bd0_b239_68213b8b8e36.slice/crio-69234490418b26bb2b1b6406652a8f1725432c9c85cfc91b8a8fba724abb50ce WatchSource:0}: Error finding container 69234490418b26bb2b1b6406652a8f1725432c9c85cfc91b8a8fba724abb50ce: Status 404 returned error can't find the container with id 69234490418b26bb2b1b6406652a8f1725432c9c85cfc91b8a8fba724abb50ce
Oct 30 17:20:29.910511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:29.910429    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2" event={"ID":"589a8b96-db1d-4bd0-b239-68213b8b8e36","Type":"ContainerStarted","Data":"69234490418b26bb2b1b6406652a8f1725432c9c85cfc91b8a8fba724abb50ce"}
Oct 30 17:20:32.502765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:32.502732    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:32.617849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:32.617810    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:33.930988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:33.930950    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2" event={"ID":"589a8b96-db1d-4bd0-b239-68213b8b8e36","Type":"ContainerStarted","Data":"f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319"}
Oct 30 17:20:33.931426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:33.931079    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:34.000978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:34.000911    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2" podStartSLOduration=2.581104455 podStartE2EDuration="6.000895138s" totalImagesPullingTime="3.419790683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:33.988550383 +0000 UTC m=+439.750594196" watchObservedRunningTime="2026-10-30 17:20:34.000895138 +0000 UTC m=+439.762938952"
Oct 30 17:20:36.662798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:36.662761    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:36.775724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:36.775686    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:37.261439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:37.261410    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:37.384973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:37.384943    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:41.071735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.071699    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:41.186406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.186371    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:41.389876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.389845    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"]
Oct 30 17:20:41.390242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.390207    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" containerName="manager" containerID="cri-o://f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319" gracePeriod=2
Oct 30 17:20:41.390708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.390684    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"]
Oct 30 17:20:41.403735 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:41.403683    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:41.645170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.645113    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:41.658460 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:41.658428    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:41.746124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.746087    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d\" (UniqueName: \"kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d\") pod \"589a8b96-db1d-4bd0-b239-68213b8b8e36\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") "
Oct 30 17:20:41.746318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.746243    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume\") pod \"589a8b96-db1d-4bd0-b239-68213b8b8e36\" (UID: \"589a8b96-db1d-4bd0-b239-68213b8b8e36\") "
Oct 30 17:20:41.746779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.746737    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume" pod "589a8b96-db1d-4bd0-b239-68213b8b8e36" (UID: "589a8b96-db1d-4bd0-b239-68213b8b8e36"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:41.748460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.748438    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d" pod "589a8b96-db1d-4bd0-b239-68213b8b8e36" (UID: "589a8b96-db1d-4bd0-b239-68213b8b8e36"). InnerVolumeSpecName "kube-api-access-wrc7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:41.846946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.846907    5534 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/589a8b96-db1d-4bd0-b239-68213b8b8e36-extensions-socket-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.846946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.846940    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrc7d\" (UniqueName: \"kubernetes.io/projected/589a8b96-db1d-4bd0-b239-68213b8b8e36-kube-api-access-wrc7d\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.962011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.961919    5534 generic.go:356] "Generic (PLEG): container finished" podID="589a8b96-db1d-4bd0-b239-68213b8b8e36" containerID="f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319" exitCode=0
Oct 30 17:20:41.962011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.961966    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:41.962191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.962016    5534 scope.go:122] "RemoveContainer" containerID="f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319"
Oct 30 17:20:41.972092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.972074    5534 scope.go:122] "RemoveContainer" containerID="f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319"
Oct 30 17:20:41.972400 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:41.972373    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319\": container with ID starting with f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319 not found: ID does not exist" containerID="f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319"
Oct 30 17:20:41.972479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:41.972413    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319"} err="failed to get container status \"f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319\": rpc error: code = NotFound desc = could not find container \"f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319\": container with ID starting with f5ed344203f3b400799215ee6a85c2e5d52f75c4fd4b341c3d370b4e0f916319 not found: ID does not exist"
Oct 30 17:20:41.975423 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:41.975398    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:41.988587 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:41.988555    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bnfq2\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bnfq2"
Oct 30 17:20:42.366610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:42.366573    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" path="/var/lib/kubelet/pods/589a8b96-db1d-4bd0-b239-68213b8b8e36/volumes"
Oct 30 17:20:45.242557 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:20:45.242523    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:20:46.281302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:46.281264    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:46.398551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:46.398521    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:46.515803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:46.515772    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:53.320518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.320488    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:53.448943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.448894    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:20:53.764478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.764413    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"]
Oct 30 17:20:53.764784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.764769    5534 cpu_manager.go:414] "containerMap: removing container" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" containerName="manager"
Oct 30 17:20:53.764855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.764785    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" containerName="manager"
Oct 30 17:20:53.764855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.764854    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="589a8b96-db1d-4bd0-b239-68213b8b8e36" containerName="manager"
Oct 30 17:20:53.771658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.771610    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:53.790670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.790615    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4ptvx\""
Oct 30 17:20:53.800709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.800687    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"]
Oct 30 17:20:53.846286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.846255    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsw6f\" (UniqueName: \"kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f\") pod \"kuadrant-operator-controller-manager-57bb9bfff-g7585\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:53.846442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.846331    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-g7585\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:53.947518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.947487    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsw6f\" (UniqueName: \"kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f\") pod \"kuadrant-operator-controller-manager-57bb9bfff-g7585\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:53.947679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.947550    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-g7585\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:53.947988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.947969    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-g7585\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:53.977114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:53.977087    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsw6f\" (UniqueName: \"kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f\") pod \"kuadrant-operator-controller-manager-57bb9bfff-g7585\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:54.085299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:54.085209    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:54.411939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:54.411914    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"]
Oct 30 17:20:54.414677 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:20:54.414630    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3418abd6_5f92_4c85_a9e1_b63c95219594.slice/crio-de5402c528766269a8419abc5b5abe2b98a69b200fcc93de3cb5b03a6d02a93b WatchSource:0}: Error finding container de5402c528766269a8419abc5b5abe2b98a69b200fcc93de3cb5b03a6d02a93b: Status 404 returned error can't find the container with id de5402c528766269a8419abc5b5abe2b98a69b200fcc93de3cb5b03a6d02a93b
Oct 30 17:20:55.015083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:55.015048    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585" event={"ID":"3418abd6-5f92-4c85-a9e1-b63c95219594","Type":"ContainerStarted","Data":"e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa"}
Oct 30 17:20:55.015248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:55.015108    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585" event={"ID":"3418abd6-5f92-4c85-a9e1-b63c95219594","Type":"ContainerStarted","Data":"de5402c528766269a8419abc5b5abe2b98a69b200fcc93de3cb5b03a6d02a93b"}
Oct 30 17:20:55.015248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:55.015154    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:20:55.061370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:20:55.061316    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585" podStartSLOduration=2.061302938 podStartE2EDuration="2.061302938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:55.060886535 +0000 UTC m=+460.822930348" watchObservedRunningTime="2026-10-30 17:20:55.061302938 +0000 UTC m=+460.823346748"
Oct 30 17:21:06.022333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:06.022303    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 17:21:10.901918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:10.901882    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:11.016912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:11.016884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:13.129782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:13.129748    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:15.243727 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:21:15.243680    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:21:18.028904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:18.028870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:25.885260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:25.885228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:26.001052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.001022    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:26.117928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.117897    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:26.773250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.773211    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-7nqc8"]
Oct 30 17:21:26.776865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.776845    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:26.798243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.798213    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-7nqc8"]
Oct 30 17:21:26.799252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.799232    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wv7nq\""
Oct 30 17:21:26.805781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.805759    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrfp\" (UniqueName: \"kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp\") pod \"authorino-57f7f8659f-7nqc8\" (UID: \"f5042c49-6829-492a-8696-abef33e8e95d\") " pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:26.861546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.861517    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:26.906537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.906509    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrfp\" (UniqueName: \"kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp\") pod \"authorino-57f7f8659f-7nqc8\" (UID: \"f5042c49-6829-492a-8696-abef33e8e95d\") " pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:26.932627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.932595    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrfp\" (UniqueName: \"kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp\") pod \"authorino-57f7f8659f-7nqc8\" (UID: \"f5042c49-6829-492a-8696-abef33e8e95d\") " pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:26.976466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:26.976434    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:27.088049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:27.087984    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:27.383320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:27.383284    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-7nqc8"]
Oct 30 17:21:27.386198 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:21:27.386156    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5042c49_6829_492a_8696_abef33e8e95d.slice/crio-04976836d73c5fd90734ae8c812b75e37f9bdaddc962a959eae1268effa9cdea WatchSource:0}: Error finding container 04976836d73c5fd90734ae8c812b75e37f9bdaddc962a959eae1268effa9cdea: Status 404 returned error can't find the container with id 04976836d73c5fd90734ae8c812b75e37f9bdaddc962a959eae1268effa9cdea
Oct 30 17:21:28.147537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:28.147497    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-7nqc8" event={"ID":"f5042c49-6829-492a-8696-abef33e8e95d","Type":"ContainerStarted","Data":"04976836d73c5fd90734ae8c812b75e37f9bdaddc962a959eae1268effa9cdea"}
Oct 30 17:21:29.796293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:29.796251    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-7nqc8"]
Oct 30 17:21:30.156962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:30.156918    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-7nqc8" event={"ID":"f5042c49-6829-492a-8696-abef33e8e95d","Type":"ContainerStarted","Data":"74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6"}
Oct 30 17:21:30.194706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:30.194631    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-7nqc8" podStartSLOduration=2.387199741 podStartE2EDuration="4.194607696s" totalImagesPullingTime="1.807407955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:30.194409803 +0000 UTC m=+495.956453616" watchObservedRunningTime="2026-10-30 17:21:30.194607696 +0000 UTC m=+495.956651701"
Oct 30 17:21:31.161741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:31.161698    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-7nqc8" podUID="f5042c49-6829-492a-8696-abef33e8e95d" containerName="authorino" containerID="cri-o://74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6" gracePeriod=30
Oct 30 17:21:31.406894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:31.406866    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:31.445143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:31.445092    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp\" (UniqueName: \"kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp\") pod \"f5042c49-6829-492a-8696-abef33e8e95d\" (UID: \"f5042c49-6829-492a-8696-abef33e8e95d\") "
Oct 30 17:21:31.447587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:31.447552    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp" pod "f5042c49-6829-492a-8696-abef33e8e95d" (UID: "f5042c49-6829-492a-8696-abef33e8e95d"). InnerVolumeSpecName "kube-api-access-hvrfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:31.546591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:31.546563    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrfp\" (UniqueName: \"kubernetes.io/projected/f5042c49-6829-492a-8696-abef33e8e95d-kube-api-access-hvrfp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:21:32.166727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.166691    5534 generic.go:356] "Generic (PLEG): container finished" podID="f5042c49-6829-492a-8696-abef33e8e95d" containerID="74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6" exitCode=0
Oct 30 17:21:32.167157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.166736    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-7nqc8"
Oct 30 17:21:32.167157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.166748    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-7nqc8" event={"ID":"f5042c49-6829-492a-8696-abef33e8e95d","Type":"ContainerDied","Data":"74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6"}
Oct 30 17:21:32.167157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.166779    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-7nqc8" event={"ID":"f5042c49-6829-492a-8696-abef33e8e95d","Type":"ContainerDied","Data":"04976836d73c5fd90734ae8c812b75e37f9bdaddc962a959eae1268effa9cdea"}
Oct 30 17:21:32.167157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.166797    5534 scope.go:122] "RemoveContainer" containerID="74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6"
Oct 30 17:21:32.175594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.175578    5534 scope.go:122] "RemoveContainer" containerID="74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6"
Oct 30 17:21:32.175931 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:21:32.175912    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6\": container with ID starting with 74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6 not found: ID does not exist" containerID="74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6"
Oct 30 17:21:32.176006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.175942    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6"} err="failed to get container status \"74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6\": rpc error: code = NotFound desc = could not find container \"74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6\": container with ID starting with 74545ed70587de2be8de617b7da9979072f397784c1650d6d88209a0f06f4ac6 not found: ID does not exist"
Oct 30 17:21:32.220725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.220701    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-7nqc8"]
Oct 30 17:21:32.224635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.224614    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-7nqc8"]
Oct 30 17:21:32.367132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:32.367101    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5042c49-6829-492a-8696-abef33e8e95d" path="/var/lib/kubelet/pods/f5042c49-6829-492a-8696-abef33e8e95d/volumes"
Oct 30 17:21:34.804588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:34.804555    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:34.915471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:34.915439    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:40.092268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:40.092230    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:40.967624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:40.967589    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:45.097964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:45.097932    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:45.243750 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:21:45.243709    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:21:46.382940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:46.382911    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:58.172170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:58.172134    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:21:58.437634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:21:58.437544    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:01.882493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:01.882460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:02.000416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:02.000365    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:15.242923 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:15.242887    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:22:18.158486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:18.158445    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:23.477302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:23.477272    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:27.177833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.177755    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:27.242714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.242686    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-2pcbw"]
Oct 30 17:22:27.243161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.243142    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f5042c49-6829-492a-8696-abef33e8e95d" containerName="authorino"
Oct 30 17:22:27.243277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.243163    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5042c49-6829-492a-8696-abef33e8e95d" containerName="authorino"
Oct 30 17:22:27.243337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.243283    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5042c49-6829-492a-8696-abef33e8e95d" containerName="authorino"
Oct 30 17:22:27.246163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.246141    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.264535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.264517    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kz7qf\""
Oct 30 17:22:27.272886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.272862    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 17:22:27.272886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.272877    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 17:22:27.273031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.272942    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 17:22:27.278199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.278181    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-2pcbw"]
Oct 30 17:22:27.297749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.297725    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.297845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.297752    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.297938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.297919    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svz6l\" (UniqueName: \"kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.299021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.299002    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:27.398915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.398883    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svz6l\" (UniqueName: \"kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.399064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.398925    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.399064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.398945    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.401562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.401533    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.401653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.401573    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.412867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.412846    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:27.424315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.424297    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svz6l\" (UniqueName: \"kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l\") pod \"maas-controller-577f596567-2pcbw\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") " pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.530520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.530448    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:27.558391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.558373    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:27.639406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.639371    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bb675f547-6kmcm"]
Oct 30 17:22:27.643787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.643766    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.666485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.666459    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb675f547-6kmcm"]
Oct 30 17:22:27.701819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.701795    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjpx\" (UniqueName: \"kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.701953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.701917    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.702004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.701971    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.803451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.803359    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.803451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.803414    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.803451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.803450    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjpx\" (UniqueName: \"kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.806148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.806121    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.806148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.806140    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.827451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.827421    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjpx\" (UniqueName: \"kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx\") pod \"maas-controller-5bb675f547-6kmcm\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") " pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:27.854832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.854808    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-2pcbw"]
Oct 30 17:22:27.857377 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:22:27.857349    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e860d75_5db0_4359_a42c_2fbe2c1fc73e.slice/crio-e9448c0076f433d21cb36219a7228ec78d2fe944550d790879a9e3671897f26c WatchSource:0}: Error finding container e9448c0076f433d21cb36219a7228ec78d2fe944550d790879a9e3671897f26c: Status 404 returned error can't find the container with id e9448c0076f433d21cb36219a7228ec78d2fe944550d790879a9e3671897f26c
Oct 30 17:22:27.858588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.858573    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:22:27.956561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:27.956523    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:28.255120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:28.255095    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb675f547-6kmcm"]
Oct 30 17:22:28.257467 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:22:28.257438    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9faf67_3b4f_454a_a7cc_6d95fea1997b.slice/crio-61532881003f7784edb8bab4c492c5c28e98f551e839705188997813060db4c2 WatchSource:0}: Error finding container 61532881003f7784edb8bab4c492c5c28e98f551e839705188997813060db4c2: Status 404 returned error can't find the container with id 61532881003f7784edb8bab4c492c5c28e98f551e839705188997813060db4c2
Oct 30 17:22:28.384181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:28.384139    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2pcbw" event={"ID":"3e860d75-5db0-4359-a42c-2fbe2c1fc73e","Type":"ContainerStarted","Data":"e9448c0076f433d21cb36219a7228ec78d2fe944550d790879a9e3671897f26c"}
Oct 30 17:22:28.385315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:28.385291    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-6kmcm" event={"ID":"9f9faf67-3b4f-454a-a7cc-6d95fea1997b","Type":"ContainerStarted","Data":"61532881003f7784edb8bab4c492c5c28e98f551e839705188997813060db4c2"}
Oct 30 17:22:31.312166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.312089    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:31.417995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.417957    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2pcbw" event={"ID":"3e860d75-5db0-4359-a42c-2fbe2c1fc73e","Type":"ContainerStarted","Data":"e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"}
Oct 30 17:22:31.418197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.418076    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:31.419511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.419489    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-6kmcm" event={"ID":"9f9faf67-3b4f-454a-a7cc-6d95fea1997b","Type":"ContainerStarted","Data":"4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848"}
Oct 30 17:22:31.419630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.419615    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:31.444223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.444195    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:31.473157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.473119    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-2pcbw" podStartSLOduration=1.38818399 podStartE2EDuration="4.473104062s" totalImagesPullingTime="3.084920072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:31.472264308 +0000 UTC m=+557.234308121" watchObservedRunningTime="2026-10-30 17:22:31.473104062 +0000 UTC m=+557.235147877"
Oct 30 17:22:31.523604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:31.523560    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5bb675f547-6kmcm" podStartSLOduration=1.7172965100000002 podStartE2EDuration="4.523546751s" totalImagesPullingTime="2.806250241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:31.522563074 +0000 UTC m=+557.284606886" watchObservedRunningTime="2026-10-30 17:22:31.523546751 +0000 UTC m=+557.285590563"
Oct 30 17:22:32.967255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:32.967228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:34.711339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.711304    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"]
Oct 30 17:22:34.714879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.714858    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.734741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.734716    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 17:22:34.743563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.743536    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 17:22:34.743773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.743622    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 17:22:34.743773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.743732    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-6xd57\""
Oct 30 17:22:34.744711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.744688    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"]
Oct 30 17:22:34.760589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.760556    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spxwh\" (UniqueName: \"kubernetes.io/projected/ccff571c-a44f-447f-a36f-fbc580cf7b0a-kube-api-access-spxwh\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.760783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.760595    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccff571c-a44f-447f-a36f-fbc580cf7b0a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.760783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.760618    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.760783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.760746    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.798588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.798539    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:34.861228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.861195    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spxwh\" (UniqueName: \"kubernetes.io/projected/ccff571c-a44f-447f-a36f-fbc580cf7b0a-kube-api-access-spxwh\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.861406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.861237    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccff571c-a44f-447f-a36f-fbc580cf7b0a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.861406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.861264    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.861406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.861290    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.861406 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:34.861374    5534 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 17:22:34.861583 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:34.861392    5534 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 17:22:34.861583 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:34.861463    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccff571c-a44f-447f-a36f-fbc580cf7b0a-tls-cert podName:ccff571c-a44f-447f-a36f-fbc580cf7b0a nodeName:}" failed. No retries permitted until 2026-10-30 17:22:35.361439276 +0000 UTC m=+561.123483084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/ccff571c-a44f-447f-a36f-fbc580cf7b0a-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-4rcsr" (UID: "ccff571c-a44f-447f-a36f-fbc580cf7b0a") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 17:22:34.861583 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:34.861536    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-proxy-source podName:ccff571c-a44f-447f-a36f-fbc580cf7b0a nodeName:}" failed. No retries permitted until 2026-10-30 17:22:35.361494414 +0000 UTC m=+561.123538237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-proxy-source") pod "usage-logs-tenancy-proxy-7d94f55f4b-4rcsr" (UID: "ccff571c-a44f-447f-a36f-fbc580cf7b0a") : configmap "usage-tenancy-proxy-source" not found
Oct 30 17:22:34.862025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.862002    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.885019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.884991    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spxwh\" (UniqueName: \"kubernetes.io/projected/ccff571c-a44f-447f-a36f-fbc580cf7b0a-kube-api-access-spxwh\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:34.966355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:34.966256    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:35.366214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:35.366183    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccff571c-a44f-447f-a36f-fbc580cf7b0a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:35.366399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:35.366217    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:35.366949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:35.366928    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ccff571c-a44f-447f-a36f-fbc580cf7b0a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:35.368692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:35.368669    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccff571c-a44f-447f-a36f-fbc580cf7b0a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4rcsr\" (UID: \"ccff571c-a44f-447f-a36f-fbc580cf7b0a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:35.627815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:35.627731    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:35.926154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:35.926128    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"]
Oct 30 17:22:35.928306 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:22:35.928277    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccff571c_a44f_447f_a36f_fbc580cf7b0a.slice/crio-1400df909f6f6ed098820c82f473380473bbaef62eee252d9e3f3d251ebd04bc WatchSource:0}: Error finding container 1400df909f6f6ed098820c82f473380473bbaef62eee252d9e3f3d251ebd04bc: Status 404 returned error can't find the container with id 1400df909f6f6ed098820c82f473380473bbaef62eee252d9e3f3d251ebd04bc
Oct 30 17:22:36.442124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:36.442085    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr" event={"ID":"ccff571c-a44f-447f-a36f-fbc580cf7b0a","Type":"ContainerStarted","Data":"1400df909f6f6ed098820c82f473380473bbaef62eee252d9e3f3d251ebd04bc"}
Oct 30 17:22:41.472485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:41.472455    5534 generic.go:356] "Generic (PLEG): container finished" podID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerID="e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca" exitCode=1
Oct 30 17:22:41.472979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:41.472535    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2pcbw" event={"ID":"3e860d75-5db0-4359-a42c-2fbe2c1fc73e","Type":"ContainerDied","Data":"e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"}
Oct 30 17:22:41.473045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:41.473015    5534 scope.go:122] "RemoveContainer" containerID="e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"
Oct 30 17:22:42.427170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:42.427137    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 17:22:42.479895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:42.479850    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2pcbw" event={"ID":"3e860d75-5db0-4359-a42c-2fbe2c1fc73e","Type":"ContainerStarted","Data":"4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea"}
Oct 30 17:22:42.480300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:42.480102    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:42.516063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:42.516031    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-2pcbw"]
Oct 30 17:22:43.705864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:43.705826    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:22:44.489333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.489287    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-2pcbw" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager" containerID="cri-o://4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea" gracePeriod=30
Oct 30 17:22:44.767498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.767455    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:44.858586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.858556    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert\") pod \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") "
Oct 30 17:22:44.858817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.858656    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l\" (UniqueName: \"kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l\") pod \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") "
Oct 30 17:22:44.858817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.858713    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs\") pod \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\" (UID: \"3e860d75-5db0-4359-a42c-2fbe2c1fc73e\") "
Oct 30 17:22:44.861535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.861499    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert" pod "3e860d75-5db0-4359-a42c-2fbe2c1fc73e" (UID: "3e860d75-5db0-4359-a42c-2fbe2c1fc73e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:44.862331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.862297    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs" pod "3e860d75-5db0-4359-a42c-2fbe2c1fc73e" (UID: "3e860d75-5db0-4359-a42c-2fbe2c1fc73e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:44.862471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.862443    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l" pod "3e860d75-5db0-4359-a42c-2fbe2c1fc73e" (UID: "3e860d75-5db0-4359-a42c-2fbe2c1fc73e"). InnerVolumeSpecName "kube-api-access-svz6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:44.960037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.960000    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:22:44.960037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.960033    5534 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-webhook-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:22:44.960037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:44.960042    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svz6l\" (UniqueName: \"kubernetes.io/projected/3e860d75-5db0-4359-a42c-2fbe2c1fc73e-kube-api-access-svz6l\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:22:45.248801 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:45.248756    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:22:45.495479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.495443    5534 generic.go:356] "Generic (PLEG): container finished" podID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerID="4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea" exitCode=0
Oct 30 17:22:45.495681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.495505    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2pcbw"
Oct 30 17:22:45.495681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.495529    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2pcbw" event={"ID":"3e860d75-5db0-4359-a42c-2fbe2c1fc73e","Type":"ContainerDied","Data":"4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea"}
Oct 30 17:22:45.495681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.495601    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2pcbw" event={"ID":"3e860d75-5db0-4359-a42c-2fbe2c1fc73e","Type":"ContainerDied","Data":"e9448c0076f433d21cb36219a7228ec78d2fe944550d790879a9e3671897f26c"}
Oct 30 17:22:45.495681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.495633    5534 scope.go:122] "RemoveContainer" containerID="4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea"
Oct 30 17:22:45.506791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.506774    5534 scope.go:122] "RemoveContainer" containerID="e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"
Oct 30 17:22:45.514668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.514626    5534 scope.go:122] "RemoveContainer" containerID="4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea"
Oct 30 17:22:45.514928 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:45.514903    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea\": container with ID starting with 4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea not found: ID does not exist" containerID="4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea"
Oct 30 17:22:45.515011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.514936    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea"} err="failed to get container status \"4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea\": rpc error: code = NotFound desc = could not find container \"4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea\": container with ID starting with 4bbbae74cfe6b538a129cdc7e3d1090e20d400dccd5cf0c9a21426c23db927ea not found: ID does not exist"
Oct 30 17:22:45.515011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.514962    5534 scope.go:122] "RemoveContainer" containerID="e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"
Oct 30 17:22:45.515248 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:22:45.515231    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca\": container with ID starting with e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca not found: ID does not exist" containerID="e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"
Oct 30 17:22:45.515317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.515252    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca"} err="failed to get container status \"e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca\": rpc error: code = NotFound desc = could not find container \"e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca\": container with ID starting with e5da535b2ee26e2aae6ec9ce146f14402a43babec0b25fe6f6d2479cd3269fca not found: ID does not exist"
Oct 30 17:22:45.548549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.548526    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-2pcbw"]
Oct 30 17:22:45.560753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:45.560721    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-2pcbw"]
Oct 30 17:22:46.367438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:46.367409    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" path="/var/lib/kubelet/pods/3e860d75-5db0-4359-a42c-2fbe2c1fc73e/volumes"
Oct 30 17:22:46.501326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:46.501291    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr" event={"ID":"ccff571c-a44f-447f-a36f-fbc580cf7b0a","Type":"ContainerStarted","Data":"25b872b8993a9f60244e60780fde759c3822f6466f32d484eaa5d9e6dfc7f470"}
Oct 30 17:22:46.501503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:46.501381    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:22:46.542468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:46.542426    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr" podStartSLOduration=2.518949675 podStartE2EDuration="12.542412562s" totalImagesPullingTime="10.023462887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:46.541612848 +0000 UTC m=+572.303656663" watchObservedRunningTime="2026-10-30 17:22:46.542412562 +0000 UTC m=+572.304456375"
Oct 30 17:22:57.512047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:22:57.512017    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4rcsr"
Oct 30 17:23:03.536100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:03.536065    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:03.654256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:03.654220    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:04.875931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:04.875890    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:04.993317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:04.993285    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:05.109159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:05.109099    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:15.243390 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:23:15.243348    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:23:18.286363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:18.286332    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:28.540303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:28.540266    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:28.661046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:28.661008    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:33.085737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:33.085700    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:45.242901 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:23:45.242858    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:23:49.279061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:49.279022    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:23:49.397694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:23:49.397665    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:08.130027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.129837    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"]
Oct 30 17:24:08.131174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.131146    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager"
Oct 30 17:24:08.131174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.131175    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager"
Oct 30 17:24:08.131325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.131200    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager"
Oct 30 17:24:08.131325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.131208    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager"
Oct 30 17:24:08.131325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.131308    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager"
Oct 30 17:24:08.131325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.131323    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e860d75-5db0-4359-a42c-2fbe2c1fc73e" containerName="manager"
Oct 30 17:24:08.134367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.134346    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.149587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.149563    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 17:24:08.161152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.161121    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 17:24:08.161273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.161257    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hbtff\""
Oct 30 17:24:08.161345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.161332    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 17:24:08.162563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.162541    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"]
Oct 30 17:24:08.169401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.169378    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbcxl\" (UniqueName: \"kubernetes.io/projected/5217b9b6-dc38-4b7e-bf14-0206e45811e6-kube-api-access-qbcxl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.169504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.169428    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.169556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.169509    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5217b9b6-dc38-4b7e-bf14-0206e45811e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.169606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.169592    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.169671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.169617    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.169716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.169698    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.215902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.215860    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:08.270969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.270938    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5217b9b6-dc38-4b7e-bf14-0206e45811e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271001    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271026    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271056    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271088    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbcxl\" (UniqueName: \"kubernetes.io/projected/5217b9b6-dc38-4b7e-bf14-0206e45811e6-kube-api-access-qbcxl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271110    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271479    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271518    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.271623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.271568    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.273517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.273496    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5217b9b6-dc38-4b7e-bf14-0206e45811e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.273857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.273839    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5217b9b6-dc38-4b7e-bf14-0206e45811e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.294441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.294422    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbcxl\" (UniqueName: \"kubernetes.io/projected/5217b9b6-dc38-4b7e-bf14-0206e45811e6-kube-api-access-qbcxl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj\" (UID: \"5217b9b6-dc38-4b7e-bf14-0206e45811e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.451368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.451281    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:08.750335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.750310    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"]
Oct 30 17:24:08.752346 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:24:08.752313    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5217b9b6_dc38_4b7e_bf14_0206e45811e6.slice/crio-6d85fce13830b0e2ac7bcd22470f79ff3852643d713a66776f163bbcac1b7f01 WatchSource:0}: Error finding container 6d85fce13830b0e2ac7bcd22470f79ff3852643d713a66776f163bbcac1b7f01: Status 404 returned error can't find the container with id 6d85fce13830b0e2ac7bcd22470f79ff3852643d713a66776f163bbcac1b7f01
Oct 30 17:24:08.834371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:08.834343    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj" event={"ID":"5217b9b6-dc38-4b7e-bf14-0206e45811e6","Type":"ContainerStarted","Data":"6d85fce13830b0e2ac7bcd22470f79ff3852643d713a66776f163bbcac1b7f01"}
Oct 30 17:24:10.051677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.051626    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"]
Oct 30 17:24:10.055135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.055108    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.070214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.070182    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 17:24:10.083452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.083431    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"]
Oct 30 17:24:10.089815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.089789    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhlx\" (UniqueName: \"kubernetes.io/projected/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-kube-api-access-tdhlx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.089950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.089819    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.089950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.089864    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.089950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.089947    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.090123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.090055    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.090123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.090105    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.130115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.130082    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:10.191626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.191569    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhlx\" (UniqueName: \"kubernetes.io/projected/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-kube-api-access-tdhlx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.191626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.191612    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.191865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.191691    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.191865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.191714    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.191865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.191750    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.191865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.191768    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.192213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.192188    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.192430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.192406    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.192500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.192436    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.194305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.194284    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.194728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.194709    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.214803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.214746    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhlx\" (UniqueName: \"kubernetes.io/projected/ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e-kube-api-access-tdhlx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5kxv\" (UID: \"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.371697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.371671    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:10.672110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.672085    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"]
Oct 30 17:24:10.674149 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:24:10.674122    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2b2b6d_7436_4a24_85ad_d81bf1b0b79e.slice/crio-51857ef232a4358738e1d7439404418c0e497e4e32aff496e947ef8a9df9326f WatchSource:0}: Error finding container 51857ef232a4358738e1d7439404418c0e497e4e32aff496e947ef8a9df9326f: Status 404 returned error can't find the container with id 51857ef232a4358738e1d7439404418c0e497e4e32aff496e947ef8a9df9326f
Oct 30 17:24:10.844114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.844078    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj" event={"ID":"5217b9b6-dc38-4b7e-bf14-0206e45811e6","Type":"ContainerStarted","Data":"9f4db7f47d9ddcf3ad44712ef950267aa23b88a9a2a57fccc041e8386f9b4d0f"}
Oct 30 17:24:10.845284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.845256    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv" event={"ID":"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e","Type":"ContainerStarted","Data":"51857ef232a4358738e1d7439404418c0e497e4e32aff496e947ef8a9df9326f"}
Oct 30 17:24:10.886627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:10.886579    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj" podStartSLOduration=1.690264804 podStartE2EDuration="2.886565224s" totalImagesPullingTime="1.19630042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:10.886497649 +0000 UTC m=+656.648541462" watchObservedRunningTime="2026-10-30 17:24:10.886565224 +0000 UTC m=+656.648609037"
Oct 30 17:24:11.452174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:11.452140    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:11.823755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:11.823662    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:11.851057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:11.851024    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv" event={"ID":"ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e","Type":"ContainerStarted","Data":"331b2983026ed226d3a2260bd1b7c16f4753b22f6c793856b87d6a1b7fadc851"}
Oct 30 17:24:11.893013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:11.892970    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv" podStartSLOduration=1.66406053 podStartE2EDuration="1.892956042s" totalImagesPullingTime="228.895512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:11.891931024 +0000 UTC m=+657.653974836" watchObservedRunningTime="2026-10-30 17:24:11.892956042 +0000 UTC m=+657.654999855"
Oct 30 17:24:12.372207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:12.372180    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:14.588970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:14.588940    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:15.246297 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:24:15.246258    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:24:17.373870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.373838    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"]
Oct 30 17:24:17.377598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.377571    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.392570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.392545    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 17:24:17.404807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.404781    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"]
Oct 30 17:24:17.443869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.443838    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:17.453007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.452974    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cac9387-6cfc-4ded-8041-00e436212e6b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.453121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.453022    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.453121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.453051    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg9nt\" (UniqueName: \"kubernetes.io/projected/8cac9387-6cfc-4ded-8041-00e436212e6b-kube-api-access-vg9nt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.453229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.453201    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.453361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.453344    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.453428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.453385    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554437    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554539    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554572    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554608    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cac9387-6cfc-4ded-8041-00e436212e6b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554667    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554705    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg9nt\" (UniqueName: \"kubernetes.io/projected/8cac9387-6cfc-4ded-8041-00e436212e6b-kube-api-access-vg9nt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.554989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.554965    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.555050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.555026    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.555169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.555137    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.557061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.557028    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cac9387-6cfc-4ded-8041-00e436212e6b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.557317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.557301    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cac9387-6cfc-4ded-8041-00e436212e6b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.579917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.579888    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg9nt\" (UniqueName: \"kubernetes.io/projected/8cac9387-6cfc-4ded-8041-00e436212e6b-kube-api-access-vg9nt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6\" (UID: \"8cac9387-6cfc-4ded-8041-00e436212e6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.692368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.692289    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:17.995979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:17.995954    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"]
Oct 30 17:24:17.998265 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:24:17.998238    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cac9387_6cfc_4ded_8041_00e436212e6b.slice/crio-ff27b70699680b000eb0e8984c3841761cde7059c2a87a1184e63cdc0601cb04 WatchSource:0}: Error finding container ff27b70699680b000eb0e8984c3841761cde7059c2a87a1184e63cdc0601cb04: Status 404 returned error can't find the container with id ff27b70699680b000eb0e8984c3841761cde7059c2a87a1184e63cdc0601cb04
Oct 30 17:24:18.421819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.421786    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:18.451560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.451530    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:18.464340 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.464315    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:18.880926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.880894    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6" event={"ID":"8cac9387-6cfc-4ded-8041-00e436212e6b","Type":"ContainerStarted","Data":"54a90c9b4331f6365905435592507dc684c108b517ec9cdad2cfa5631ba8a0a4"}
Oct 30 17:24:18.881128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.880946    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6" event={"ID":"8cac9387-6cfc-4ded-8041-00e436212e6b","Type":"ContainerStarted","Data":"ff27b70699680b000eb0e8984c3841761cde7059c2a87a1184e63cdc0601cb04"}
Oct 30 17:24:18.892515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.892483    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj"
Oct 30 17:24:18.924323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:18.924267    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6" podStartSLOduration=1.695917278 podStartE2EDuration="1.924248551s" totalImagesPullingTime="228.331273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:18.923801778 +0000 UTC m=+664.685845593" watchObservedRunningTime="2026-10-30 17:24:18.924248551 +0000 UTC m=+664.686292365"
Oct 30 17:24:19.037476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:19.037449    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:19.692946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:19.692909    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:20.175745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.175711    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"]
Oct 30 17:24:20.180076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.180043    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.197000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.196977    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 17:24:20.209660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.209623    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"]
Oct 30 17:24:20.240933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.240903    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:20.281080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.281044    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.281228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.281113    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.281228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.281198    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmc6\" (UniqueName: \"kubernetes.io/projected/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-kube-api-access-7gmc6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.281362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.281340    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.281400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.281375    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.281496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.281477    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.372536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.372499    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:20.382101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382072    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382113    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382172    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382233    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382279    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382304    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmc6\" (UniqueName: \"kubernetes.io/projected/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-kube-api-access-7gmc6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382686    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.382876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382855    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.383019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.382947    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.385293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.385270    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.386963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.386937    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:20.387890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.387867    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.405535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.405505    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmc6\" (UniqueName: \"kubernetes.io/projected/e3ec1506-9ff5-48f6-9bd7-d19a6a095892-kube-api-access-7gmc6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx\" (UID: \"e3ec1506-9ff5-48f6-9bd7-d19a6a095892\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.492478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.492396    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:20.793771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.793745    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"]
Oct 30 17:24:20.795320 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:24:20.795293    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3ec1506_9ff5_48f6_9bd7_d19a6a095892.slice/crio-a6a7e8be92b16ae1018ded8de3c14abe4d52216181cd557da20826b59d20adc4 WatchSource:0}: Error finding container a6a7e8be92b16ae1018ded8de3c14abe4d52216181cd557da20826b59d20adc4: Status 404 returned error can't find the container with id a6a7e8be92b16ae1018ded8de3c14abe4d52216181cd557da20826b59d20adc4
Oct 30 17:24:20.898328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.898293    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" event={"ID":"e3ec1506-9ff5-48f6-9bd7-d19a6a095892","Type":"ContainerStarted","Data":"a6a7e8be92b16ae1018ded8de3c14abe4d52216181cd557da20826b59d20adc4"}
Oct 30 17:24:20.909634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:20.909614    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv"
Oct 30 17:24:21.905089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:21.905049    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" event={"ID":"e3ec1506-9ff5-48f6-9bd7-d19a6a095892","Type":"ContainerStarted","Data":"bcb694a2163e601158dd0e1e793f23b2bc3e4b6a85eeec182325c0fbfaa7f7d1"}
Oct 30 17:24:21.948765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:21.948690    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podStartSLOduration=1.7280848359999998 podStartE2EDuration="1.94867031s" totalImagesPullingTime="220.585474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:21.947935149 +0000 UTC m=+667.709978963" watchObservedRunningTime="2026-10-30 17:24:21.94867031 +0000 UTC m=+667.710714125"
Oct 30 17:24:22.493377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:22.493338    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:25.585384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:25.585349    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:25.696043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:25.696010    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:26.845750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.845716    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"]
Oct 30 17:24:26.848868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.848852    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.882295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.882269    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 17:24:26.882600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.882581    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mccnk\""
Oct 30 17:24:26.890979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.890958    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"]
Oct 30 17:24:26.895921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.895897    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"]
Oct 30 17:24:26.899906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.899881    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:26.932044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.932018    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"]
Oct 30 17:24:26.944314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944292    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944338    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944443    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944536    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8r9\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944666    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944705    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944803    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944856    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:26.944964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:26.944936    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.045472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045440    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.045472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045477    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.045798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045511    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.045798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045599    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.045798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045681    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8r9\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.045916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045795    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.045916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045876    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.045996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045923    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.045996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045943    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.045996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045965    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglpx\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.045996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.045988    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046007    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046065    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.046211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046093    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.046211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046121    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046147    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.046211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046195    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046233    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046296    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046566    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046607    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046827    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.046906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.046840    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.048163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.048148    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.048754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.048737    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.068934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.068904    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.069192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.069172    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8r9\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9\") pod \"maas-default-gateway-openshift-default-7df854bf67-lmw9v\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.147047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.146986    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147029    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147057    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147123    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147189    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147233    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147267    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147290    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147323    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglpx\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147501    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147832    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147855    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.147984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.147956    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.148297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.148276    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.149818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.149792    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.149940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.149922    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.161840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.161817    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:27.170114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.170085    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.170200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.170184    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglpx\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx\") pod \"maas-default-gateway-openshift-default-7df854bf67-tbsxh\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.215007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.214962    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:27.492819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.491415    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"]
Oct 30 17:24:27.529838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.529814    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"]
Oct 30 17:24:27.532235 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:24:27.532209    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2448eac0_836f_4471_ae1a_f9fa9c7c45ff.slice/crio-9b97cfa9f4b319c3006280bd7174cb4a7ebc0f29b12f91a46fa5c3fd83d0f9a0 WatchSource:0}: Error finding container 9b97cfa9f4b319c3006280bd7174cb4a7ebc0f29b12f91a46fa5c3fd83d0f9a0: Status 404 returned error can't find the container with id 9b97cfa9f4b319c3006280bd7174cb4a7ebc0f29b12f91a46fa5c3fd83d0f9a0
Oct 30 17:24:27.693111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.693082    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:27.705769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.705747    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:27.932208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.932172    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" event={"ID":"2448eac0-836f-4471-ae1a-f9fa9c7c45ff","Type":"ContainerStarted","Data":"9b97cfa9f4b319c3006280bd7174cb4a7ebc0f29b12f91a46fa5c3fd83d0f9a0"}
Oct 30 17:24:27.933390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.933364    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" event={"ID":"e5a9f812-5e0d-4fb8-ab84-77d523901927","Type":"ContainerStarted","Data":"a9c7de232679c6f6744d64d3e2f15e6b1652c49cfa3270f713bf1e18e25c30d2"}
Oct 30 17:24:27.944472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:27.944451    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6"
Oct 30 17:24:30.218540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.218509    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:24:30.218860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.218572    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:24:30.218860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.218602    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:24:30.493438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.493360    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:30.505801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.505775    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:30.949592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.949558    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" event={"ID":"e5a9f812-5e0d-4fb8-ab84-77d523901927","Type":"ContainerStarted","Data":"8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481"}
Oct 30 17:24:30.988845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:30.988816    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:24:31.002981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:31.002934    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podStartSLOduration=2.285607507 podStartE2EDuration="5.002920712s" totalImagesPullingTime="2.717313205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:31.002421127 +0000 UTC m=+676.764464964" watchObservedRunningTime="2026-10-30 17:24:31.002920712 +0000 UTC m=+676.764964524"
Oct 30 17:24:31.177287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:31.176802    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:31.177709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:31.177543    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:31.177709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:31.177606    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:32.170668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:32.165976    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:32.170668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:32.166033    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:33.177679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.166985    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:33.177679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.167049    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:33.219751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.219717    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:33.824497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.824461    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bc8c78ff8-kx77f"]
Oct 30 17:24:33.828991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.828968    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:33.848332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.848305    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bc8c78ff8-kx77f"]
Oct 30 17:24:33.860624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.860568    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 17:24:33.860789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.860769    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wv7nq\""
Oct 30 17:24:33.914725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.914691    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb838fb8-f499-4b7f-a2ff-99c126affe86-tls-cert\") pod \"authorino-5bc8c78ff8-kx77f\" (UID: \"cb838fb8-f499-4b7f-a2ff-99c126affe86\") " pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:33.914873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:33.914745    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25gk\" (UniqueName: \"kubernetes.io/projected/cb838fb8-f499-4b7f-a2ff-99c126affe86-kube-api-access-m25gk\") pod \"authorino-5bc8c78ff8-kx77f\" (UID: \"cb838fb8-f499-4b7f-a2ff-99c126affe86\") " pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:34.015694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.015660    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb838fb8-f499-4b7f-a2ff-99c126affe86-tls-cert\") pod \"authorino-5bc8c78ff8-kx77f\" (UID: \"cb838fb8-f499-4b7f-a2ff-99c126affe86\") " pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:34.015873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.015729    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25gk\" (UniqueName: \"kubernetes.io/projected/cb838fb8-f499-4b7f-a2ff-99c126affe86-kube-api-access-m25gk\") pod \"authorino-5bc8c78ff8-kx77f\" (UID: \"cb838fb8-f499-4b7f-a2ff-99c126affe86\") " pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:34.018164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.018140    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb838fb8-f499-4b7f-a2ff-99c126affe86-tls-cert\") pod \"authorino-5bc8c78ff8-kx77f\" (UID: \"cb838fb8-f499-4b7f-a2ff-99c126affe86\") " pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:34.039394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.039366    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25gk\" (UniqueName: \"kubernetes.io/projected/cb838fb8-f499-4b7f-a2ff-99c126affe86-kube-api-access-m25gk\") pod \"authorino-5bc8c78ff8-kx77f\" (UID: \"cb838fb8-f499-4b7f-a2ff-99c126affe86\") " pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:34.140789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.140759    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc8c78ff8-kx77f"
Oct 30 17:24:34.177917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.176675    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:34.177917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.176732    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:34.311602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.311575    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:34.431683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.431576    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:34.476678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.473741    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bc8c78ff8-kx77f"]
Oct 30 17:24:34.994533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:34.994500    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc8c78ff8-kx77f" event={"ID":"cb838fb8-f499-4b7f-a2ff-99c126affe86","Type":"ContainerStarted","Data":"ba706849dea52dab49d8f766e26fa21b4e400216e3ec4382d09a2806fbd2fa42"}
Oct 30 17:24:35.167675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:35.166990    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:35.167675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:35.167051    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:36.162707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:36.162673    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:36.163088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:36.162740    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:36.702844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:36.702772    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:24:36.702844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:36.702844    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:24:36.703041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:36.702870    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:24:37.008152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.008057    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" event={"ID":"2448eac0-836f-4471-ae1a-f9fa9c7c45ff","Type":"ContainerStarted","Data":"c91a25f3f6af478d17dd47185d3246bd3e970dee697dcfa6c3c6616a2028ee43"}
Oct 30 17:24:37.010274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.010228    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc8c78ff8-kx77f" event={"ID":"cb838fb8-f499-4b7f-a2ff-99c126affe86","Type":"ContainerStarted","Data":"01c5ae8d7a6ac3adc6e54706b7b5685bf9bfcef93ac53b08ddc35394917e0a03"}
Oct 30 17:24:37.068445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.068376    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podStartSLOduration=1.900420735 podStartE2EDuration="11.068357025s" totalImagesPullingTime="9.16793629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:37.067200851 +0000 UTC m=+682.829244664" watchObservedRunningTime="2026-10-30 17:24:37.068357025 +0000 UTC m=+682.830400840"
Oct 30 17:24:37.107294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.107235    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5bc8c78ff8-kx77f" podStartSLOduration=2.326871315 podStartE2EDuration="4.107219081s" totalImagesPullingTime="1.780347766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:37.106172551 +0000 UTC m=+682.868216364" watchObservedRunningTime="2026-10-30 17:24:37.107219081 +0000 UTC m=+682.869262926"
Oct 30 17:24:37.168631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.168043    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 17:24:37.168631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.168117    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 17:24:37.226700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.223836    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:37.230242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.230210    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:37.230498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:37.230468    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:38.215447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:38.215410    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:38.215990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:38.215478    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:38.812341 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:38.812285    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:38.969794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:38.969762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:24:39.183665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:39.180748    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:24:39.183665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:39.180815    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:24:39.215438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:39.215408    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:39.215601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:39.215465    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:40.182788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:40.182750    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:24:40.183104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:40.183068    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:24:40.216166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:40.216132    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:40.216533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:40.216190    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:41.216359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.216317    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:41.216928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.216388    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:41.229824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.229777    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:24:41.229972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.229843    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:24:41.952142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.950948    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:41.952142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.952090    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:41.955666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:41.952784    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 17:24:42.216055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:42.215967    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:42.216257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:42.216235    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:42.216339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:42.216292    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:43.215959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:43.215916    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:43.216504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:43.215992    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:44.216360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:44.216311    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 17:24:44.216878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:44.216377    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 17:24:45.245272 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:24:45.245231    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:24:46.256828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:46.256792    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:24:46.257342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:46.256865    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:24:47.298838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:47.298793    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:24:47.298838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:47.298859    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:24:47.335058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:47.335029    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:47.336382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:47.336358    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:24:49.724612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.724582    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-59c45c7574-t8dth"]
Oct 30 17:24:49.730797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.730773    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:49.749448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.749416    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-59c45c7574-t8dth"]
Oct 30 17:24:49.759527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.759502    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 17:24:49.759527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.759512    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-x52p7\""
Oct 30 17:24:49.759746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.759531    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 17:24:49.917310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.917278    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd0270f0-f92b-4192-9b07-fb82620137f0-metrics-certs\") pod \"ai-gateway-controller-59c45c7574-t8dth\" (UID: \"fd0270f0-f92b-4192-9b07-fb82620137f0\") " pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:49.917464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:49.917392    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggqg\" (UniqueName: \"kubernetes.io/projected/fd0270f0-f92b-4192-9b07-fb82620137f0-kube-api-access-pggqg\") pod \"ai-gateway-controller-59c45c7574-t8dth\" (UID: \"fd0270f0-f92b-4192-9b07-fb82620137f0\") " pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:50.018760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:50.018689    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggqg\" (UniqueName: \"kubernetes.io/projected/fd0270f0-f92b-4192-9b07-fb82620137f0-kube-api-access-pggqg\") pod \"ai-gateway-controller-59c45c7574-t8dth\" (UID: \"fd0270f0-f92b-4192-9b07-fb82620137f0\") " pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:50.018893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:50.018761    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd0270f0-f92b-4192-9b07-fb82620137f0-metrics-certs\") pod \"ai-gateway-controller-59c45c7574-t8dth\" (UID: \"fd0270f0-f92b-4192-9b07-fb82620137f0\") " pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:50.021462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:50.021440    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd0270f0-f92b-4192-9b07-fb82620137f0-metrics-certs\") pod \"ai-gateway-controller-59c45c7574-t8dth\" (UID: \"fd0270f0-f92b-4192-9b07-fb82620137f0\") " pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:50.042156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:50.042128    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggqg\" (UniqueName: \"kubernetes.io/projected/fd0270f0-f92b-4192-9b07-fb82620137f0-kube-api-access-pggqg\") pod \"ai-gateway-controller-59c45c7574-t8dth\" (UID: \"fd0270f0-f92b-4192-9b07-fb82620137f0\") " pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:50.043751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:50.043733    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:50.342250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:50.342230    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-59c45c7574-t8dth"]
Oct 30 17:24:50.344245 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:24:50.344216    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0270f0_f92b_4192_9b07_fb82620137f0.slice/crio-8a07624bfd6d15a5b3043a67240eb2aace1da873922e05240cc4c7a36ca8eb1a WatchSource:0}: Error finding container 8a07624bfd6d15a5b3043a67240eb2aace1da873922e05240cc4c7a36ca8eb1a: Status 404 returned error can't find the container with id 8a07624bfd6d15a5b3043a67240eb2aace1da873922e05240cc4c7a36ca8eb1a
Oct 30 17:24:51.110424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:51.110374    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth" event={"ID":"fd0270f0-f92b-4192-9b07-fb82620137f0","Type":"ContainerStarted","Data":"8a07624bfd6d15a5b3043a67240eb2aace1da873922e05240cc4c7a36ca8eb1a"}
Oct 30 17:24:53.120687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:53.120628    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth" event={"ID":"fd0270f0-f92b-4192-9b07-fb82620137f0","Type":"ContainerStarted","Data":"ca8f080f6349747306bc7564011a05a17ce7e93afd963e498e81781ec86fe38c"}
Oct 30 17:24:53.121123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:53.120785    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:24:53.160984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:24:53.160942    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth" podStartSLOduration=1.963781327 podStartE2EDuration="4.160929223s" totalImagesPullingTime="2.197147896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:53.160313418 +0000 UTC m=+698.922357235" watchObservedRunningTime="2026-10-30 17:24:53.160929223 +0000 UTC m=+698.922973035"
Oct 30 17:25:04.127226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:04.127190    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-59c45c7574-t8dth"
Oct 30 17:25:15.242669 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:25:15.242599    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:25:15.951304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:15.951252    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:16.523384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:16.523347    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:18.240653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:18.240601    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:18.356279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:18.356247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:19.829490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:19.829453    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:19.980881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:19.980848    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:33.345241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:25:33.345127    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:25:45.243119 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:25:45.243082    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:26:15.243922 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:26:15.243882    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:26:21.449572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.449540    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:21.565762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.565716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:21.884953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.884919    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"]
Oct 30 17:26:21.888581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.888546    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:21.909353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.909327    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-27567a-openshift-default-dockercfg-cnckb\""
Oct 30 17:26:21.920951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.920915    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"]
Oct 30 17:26:21.944706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:21.944668    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.058630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.058592    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.058820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.058674    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.058820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.058738    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.058820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.058807    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.058994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.058880    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.058994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.058929    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.059072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.059050    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnzc\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.059177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.059161    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.059242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.059194    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.086447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.086407    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.160240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160161    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160193    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160247    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160268    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160282    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160296    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160346    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160391    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160449    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnzc\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160673    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160722    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.160961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.160868    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.161247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.161222    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.161394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.161267    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.163357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.163331    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.163540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.163519    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.167761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.167739    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"]
Oct 30 17:26:22.171951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.171927    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.184372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.184345    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.184998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.184973    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnzc\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.186563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.186541    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-455086-openshift-default-dockercfg-98bfj\""
Oct 30 17:26:22.200335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.200310    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"]
Oct 30 17:26:22.209445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.208483    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:22.217389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.217346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.261311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261281    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261349    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skflp\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261504    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261557    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261585    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261685    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261737    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261761    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.261873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.261835    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.340910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.340862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.363213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363184    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363222    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363245    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skflp\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363279    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363299    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363317    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363366    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363409    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363433    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363813    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.363931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363905    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.364015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.363997    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.364143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.364124    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.364271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.364252    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.366203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.366164    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.366657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.366615    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.386881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.386862    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.387094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.387076    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skflp\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.455595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.455523    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"]
Oct 30 17:26:22.460396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.460377    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.467080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.467056    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.477576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.477552    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-306aed-openshift-default-dockercfg-rbbs2\""
Oct 30 17:26:22.483604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.483580    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"]
Oct 30 17:26:22.510438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.510413    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:22.536763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.536717    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"]
Oct 30 17:26:22.540943 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:22.540917    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc688455a_36a2_4f26_9ccd_20be9f3af129.slice/crio-e90a8e01f4ce2d811af80d46298d8fa7b0b3917d15120b4b28233d43ddb51d08 WatchSource:0}: Error finding container e90a8e01f4ce2d811af80d46298d8fa7b0b3917d15120b4b28233d43ddb51d08: Status 404 returned error can't find the container with id e90a8e01f4ce2d811af80d46298d8fa7b0b3917d15120b4b28233d43ddb51d08
Oct 30 17:26:22.543231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.543194    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:22.543334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.543281    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:22.543334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.543327    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:22.565231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565209    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565243    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565272    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565300    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565341    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565359    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565384    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565425    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.565552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.565478    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wff2q\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.603438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.603331    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.667074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667039    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wff2q\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667090    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667122    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667160    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667204    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667257    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667287    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667326    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667381    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667799    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667837    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.667965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.667934    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.668347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.668291    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.668621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.668435    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.670564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.670514    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.670564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.670548    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.716613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.716521    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.716803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.716635    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wff2q\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.775801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.775471    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:22.810688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.810635    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:22.833987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.833922    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"]
Oct 30 17:26:22.850669 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:22.850597    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cfe7a4_8d40_41b6_a066_a6619d20138a.slice/crio-0c5c74eecb4ecff16d66f10af38afffd7451a47ba79ff1d0656aa2d6c794c421 WatchSource:0}: Error finding container 0c5c74eecb4ecff16d66f10af38afffd7451a47ba79ff1d0656aa2d6c794c421: Status 404 returned error can't find the container with id 0c5c74eecb4ecff16d66f10af38afffd7451a47ba79ff1d0656aa2d6c794c421
Oct 30 17:26:22.855944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.855901    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:22.856049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.855997    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:22.856112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.856050    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:22.927250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:22.927226    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:23.091396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.091368    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"]
Oct 30 17:26:23.093300 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:23.093272    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683cd04d_90d2_4f93_b86d_c72d0f33bf13.slice/crio-ca4e439b14bf35520389d600717477fe451ba9bcaea93041467cc0c9a271c9e1 WatchSource:0}: Error finding container ca4e439b14bf35520389d600717477fe451ba9bcaea93041467cc0c9a271c9e1: Status 404 returned error can't find the container with id ca4e439b14bf35520389d600717477fe451ba9bcaea93041467cc0c9a271c9e1
Oct 30 17:26:23.095696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.095617    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:23.095772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.095753    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:23.095819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.095803    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:23.529852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.529750    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" event={"ID":"c688455a-36a2-4f26-9ccd-20be9f3af129","Type":"ContainerStarted","Data":"a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"}
Oct 30 17:26:23.529852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.529818    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" event={"ID":"c688455a-36a2-4f26-9ccd-20be9f3af129","Type":"ContainerStarted","Data":"e90a8e01f4ce2d811af80d46298d8fa7b0b3917d15120b4b28233d43ddb51d08"}
Oct 30 17:26:23.531522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.531489    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" event={"ID":"683cd04d-90d2-4f93-b86d-c72d0f33bf13","Type":"ContainerStarted","Data":"2a5e06a4089631dff13a86b92f407ed5790e6cf7a085c4dcc7b69e64b5af5ade"}
Oct 30 17:26:23.531692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.531542    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" event={"ID":"683cd04d-90d2-4f93-b86d-c72d0f33bf13","Type":"ContainerStarted","Data":"ca4e439b14bf35520389d600717477fe451ba9bcaea93041467cc0c9a271c9e1"}
Oct 30 17:26:23.533044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.533023    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" event={"ID":"62cfe7a4-8d40-41b6-a066-a6619d20138a","Type":"ContainerStarted","Data":"e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"}
Oct 30 17:26:23.533149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.533069    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" event={"ID":"62cfe7a4-8d40-41b6-a066-a6619d20138a","Type":"ContainerStarted","Data":"0c5c74eecb4ecff16d66f10af38afffd7451a47ba79ff1d0656aa2d6c794c421"}
Oct 30 17:26:23.551283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.551257    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:23.576045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.576003    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podStartSLOduration=2.575990347 podStartE2EDuration="2.575990347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:23.575590163 +0000 UTC m=+789.337633979" watchObservedRunningTime="2026-10-30 17:26:23.575990347 +0000 UTC m=+789.338034160"
Oct 30 17:26:23.621246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.621188    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podStartSLOduration=1.621171979 podStartE2EDuration="1.621171979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:23.620312002 +0000 UTC m=+789.382355816" watchObservedRunningTime="2026-10-30 17:26:23.621171979 +0000 UTC m=+789.383215791"
Oct 30 17:26:23.666167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.666117    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podStartSLOduration=1.666103015 podStartE2EDuration="1.666103015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:23.664537954 +0000 UTC m=+789.426581768" watchObservedRunningTime="2026-10-30 17:26:23.666103015 +0000 UTC m=+789.428146833"
Oct 30 17:26:23.776729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:23.776695    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:24.208815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.208782    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:24.213566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.213537    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:24.510877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.510783    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:24.516408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.516385    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:24.538191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.538151    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:24.538573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.538230    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:24.538763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.538738    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:24.538968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.538954    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:24.781746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:24.781669    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:25.506224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:25.506184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:25.542277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:25.542247    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:25.543216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:25.543191    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:25.645987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:25.645958    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.111686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.111631    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.243157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.243097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.376959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.376877    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.503604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.503574    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.625373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.625345    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.779180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.779103    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:26.913087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:26.913060    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:27.401453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.401423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:27.537452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.537419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:27.656377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.656299    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:27.735797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.735758    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"]
Oct 30 17:26:27.739622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.739596    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:27.755004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.754974    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-db98b0\""
Oct 30 17:26:27.766799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.766773    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-db98b0-dockercfg-cchxz\""
Oct 30 17:26:27.768911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.768887    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"]
Oct 30 17:26:27.772656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.772618    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:27.827459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.827428    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw8zn\" (UniqueName: \"kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn\") pod \"payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:27.827671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.827539    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:27.929140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.929042    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8zn\" (UniqueName: \"kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn\") pod \"payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:27.929316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.929169    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:27.929999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.929976    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:27.952153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:27.952128    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8zn\" (UniqueName: \"kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn\") pod \"payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:28.053983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:28.053953    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:28.355961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:28.355934    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"]
Oct 30 17:26:28.357978 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:28.357951    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841b5998_a749_4267_9df7_64ff045cc236.slice/crio-0c71612ab58cd1fde3c69b29fc9b5811de77cc14d61779c9b9e1a23e6f549547 WatchSource:0}: Error finding container 0c71612ab58cd1fde3c69b29fc9b5811de77cc14d61779c9b9e1a23e6f549547: Status 404 returned error can't find the container with id 0c71612ab58cd1fde3c69b29fc9b5811de77cc14d61779c9b9e1a23e6f549547
Oct 30 17:26:28.421653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:28.421603    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:28.559182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:28.559148    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" event={"ID":"841b5998-a749-4267-9df7-64ff045cc236","Type":"ContainerStarted","Data":"0c71612ab58cd1fde3c69b29fc9b5811de77cc14d61779c9b9e1a23e6f549547"}
Oct 30 17:26:29.240224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.240184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:29.345172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.345140    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"]
Oct 30 17:26:29.348891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.348865    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.361845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.361815    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:29.365148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.365124    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-27567a\""
Oct 30 17:26:29.377489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.377463    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-27567a-dockercfg-zs465\""
Oct 30 17:26:29.379887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.379858    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"]
Oct 30 17:26:29.445510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.445477    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.445727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.445549    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dk7s\" (UniqueName: \"kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s\") pod \"payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.546881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.546794    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.546881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.546854    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dk7s\" (UniqueName: \"kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s\") pod \"payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.548191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.548071    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.571138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.571115    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dk7s\" (UniqueName: \"kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s\") pod \"payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.577027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.576989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:29.665471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.665433    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:29.981430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:29.981400    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"]
Oct 30 17:26:29.982181 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:29.982149    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48c1de7_d6c3_4e39_8dc8_507b8ce7f64a.slice/crio-1e54da4152bce53ad62e18899620618ccfa5d93386c4709b735ee06f41f1eac6 WatchSource:0}: Error finding container 1e54da4152bce53ad62e18899620618ccfa5d93386c4709b735ee06f41f1eac6: Status 404 returned error can't find the container with id 1e54da4152bce53ad62e18899620618ccfa5d93386c4709b735ee06f41f1eac6
Oct 30 17:26:30.124722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.124623    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:30.241234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.241168    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:30.241583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.241505    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"]
Oct 30 17:26:30.252239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.252056    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.269256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.269230    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"]
Oct 30 17:26:30.278971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.278947    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-455086\""
Oct 30 17:26:30.278971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.278952    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-455086-dockercfg-7lv9w\""
Oct 30 17:26:30.360666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.355319    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.360666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.356713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsphk\" (UniqueName: \"kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk\") pod \"payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.458333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.458064    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsphk\" (UniqueName: \"kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk\") pod \"payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.458333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.458231    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.459183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.459149    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.483095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.483066    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsphk\" (UniqueName: \"kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk\") pod \"payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.568020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.567952    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:30.572786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.572750    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" event={"ID":"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a","Type":"ContainerStarted","Data":"1e54da4152bce53ad62e18899620618ccfa5d93386c4709b735ee06f41f1eac6"}
Oct 30 17:26:30.890415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.888502    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"]
Oct 30 17:26:30.944400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:30.944369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:31.066478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.066441    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:31.201288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.201246    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"]
Oct 30 17:26:31.204735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.204710    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.228443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.228417    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"]
Oct 30 17:26:31.237708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.237685    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-306aed\""
Oct 30 17:26:31.237881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.237862    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-306aed-dockercfg-sgvlk\""
Oct 30 17:26:31.267012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.266983    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.267538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.267181    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686cx\" (UniqueName: \"kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx\") pod \"payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.346844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.346811    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:31.367759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.367726    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686cx\" (UniqueName: \"kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx\") pod \"payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.367953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.367795    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.368608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.368581    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.393372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.393343    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686cx\" (UniqueName: \"kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx\") pod \"payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.525000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.524913    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:31.582055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.580739    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" event={"ID":"841b5998-a749-4267-9df7-64ff045cc236","Type":"ContainerStarted","Data":"c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15"}
Oct 30 17:26:31.582055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.581691    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:31.583713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.583567    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" event={"ID":"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a","Type":"ContainerStarted","Data":"6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce"}
Oct 30 17:26:31.583861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.583819    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:31.586120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.586091    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" event={"ID":"2f8f8383-1e7b-4d07-9993-3fe0e4385b02","Type":"ContainerStarted","Data":"3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84"}
Oct 30 17:26:31.586225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.586140    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" event={"ID":"2f8f8383-1e7b-4d07-9993-3fe0e4385b02","Type":"ContainerStarted","Data":"2bea7932978bd641a544018543a2488fd9c55fead863941c2073d4698fd41eff"}
Oct 30 17:26:31.586438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.586419    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:31.631220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.630986    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" podStartSLOduration=2.189544184 podStartE2EDuration="4.63096705s" totalImagesPullingTime="2.441422866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.62977254 +0000 UTC m=+797.391816353" watchObservedRunningTime="2026-10-30 17:26:31.63096705 +0000 UTC m=+797.393010864"
Oct 30 17:26:31.671684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.671616    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" podStartSLOduration=1.671602188 podStartE2EDuration="1.671602188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.670357852 +0000 UTC m=+797.432401666" watchObservedRunningTime="2026-10-30 17:26:31.671602188 +0000 UTC m=+797.433646000"
Oct 30 17:26:31.710778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.710730    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" podStartSLOduration=1.486416261 podStartE2EDuration="2.710716422s" totalImagesPullingTime="1.224300161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.70980502 +0000 UTC m=+797.471848859" watchObservedRunningTime="2026-10-30 17:26:31.710716422 +0000 UTC m=+797.472760234"
Oct 30 17:26:31.844874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:31.844838    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"]
Oct 30 17:26:31.852257 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:31.852225    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae88df68_9b94_4341_abe2_e7155ea3f29f.slice/crio-eb8cdd327786ec03641c9ccbb73d3aca8c24bc5bd8663d709630758429848f12 WatchSource:0}: Error finding container eb8cdd327786ec03641c9ccbb73d3aca8c24bc5bd8663d709630758429848f12: Status 404 returned error can't find the container with id eb8cdd327786ec03641c9ccbb73d3aca8c24bc5bd8663d709630758429848f12
Oct 30 17:26:32.020867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.020827    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:32.132950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.132840    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:32.565327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.565246    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:32.592353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.592314    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" event={"ID":"ae88df68-9b94-4341-abe2-e7155ea3f29f","Type":"ContainerStarted","Data":"38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b"}
Oct 30 17:26:32.592515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.592389    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" event={"ID":"ae88df68-9b94-4341-abe2-e7155ea3f29f","Type":"ContainerStarted","Data":"eb8cdd327786ec03641c9ccbb73d3aca8c24bc5bd8663d709630758429848f12"}
Oct 30 17:26:32.592789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.592772    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:32.633002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.632953    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" podStartSLOduration=1.6329369630000001 podStartE2EDuration="1.632936963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:32.632719574 +0000 UTC m=+798.394763387" watchObservedRunningTime="2026-10-30 17:26:32.632936963 +0000 UTC m=+798.394980775"
Oct 30 17:26:32.701811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.701776    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:32.723921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.723881    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"]
Oct 30 17:26:32.727912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.727885    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:32.748412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.748382    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"]
Oct 30 17:26:32.782700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.782662    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2z7\" (UniqueName: \"kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7\") pod \"payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:32.782900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.782720    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:32.810666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.810084    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-506124-dockercfg-rck28\""
Oct 30 17:26:32.810666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.810185    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-506124\""
Oct 30 17:26:32.883776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.883743    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2z7\" (UniqueName: \"kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7\") pod \"payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:32.883969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.883816    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:32.884598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.884571    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:32.907504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:32.907474    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2z7\" (UniqueName: \"kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7\") pod \"payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:33.042693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.042609    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:33.135721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.135617    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:33.253146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.253089    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:33.349676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.349621    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"]
Oct 30 17:26:33.351626 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:33.351599    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ad5c83_e9c0_497f_b612_496219e88040.slice/crio-67be9eb58c61892073344fd15b5df5cec9d94db469a312ad0e372023ed5e48fb WatchSource:0}: Error finding container 67be9eb58c61892073344fd15b5df5cec9d94db469a312ad0e372023ed5e48fb: Status 404 returned error can't find the container with id 67be9eb58c61892073344fd15b5df5cec9d94db469a312ad0e372023ed5e48fb
Oct 30 17:26:33.370694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.370629    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:33.598414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.598374    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" event={"ID":"a3ad5c83-e9c0-497f-b612-496219e88040","Type":"ContainerStarted","Data":"62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00"}
Oct 30 17:26:33.598877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.598437    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" event={"ID":"a3ad5c83-e9c0-497f-b612-496219e88040","Type":"ContainerStarted","Data":"67be9eb58c61892073344fd15b5df5cec9d94db469a312ad0e372023ed5e48fb"}
Oct 30 17:26:33.598877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.598659    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:33.643856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:33.643761    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" podStartSLOduration=1.6437416950000001 podStartE2EDuration="1.643741695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:33.642262485 +0000 UTC m=+799.404306299" watchObservedRunningTime="2026-10-30 17:26:33.643741695 +0000 UTC m=+799.405785508"
Oct 30 17:26:34.854854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:34.854824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:34.973414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:34.973376    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:35.087687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:35.087627    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:36.281194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:36.281138    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:36.402592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:36.402556    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:37.010384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:37.010351    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:37.134469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:37.134430    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:37.247447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:37.247406    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:38.809685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.809635    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:38.919791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.919762    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"]
Oct 30 17:26:38.923729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.923710    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:38.947391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.947368    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"]
Oct 30 17:26:38.954858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.954836    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-8d946d\""
Oct 30 17:26:38.992814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.992433    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"]
Oct 30 17:26:38.996940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.996910    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:38.997090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.997007    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpgwv\" (UniqueName: \"kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv\") pod \"payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:38.997334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.997311    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:38.997775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:38.997687    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.015378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.014896    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-8d946d-dockercfg-nvj68\""
Oct 30 17:26:39.015378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.015167    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"]
Oct 30 17:26:39.120604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.101874    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvzr\" (UniqueName: \"kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr\") pod \"payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.120604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.102033    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpgwv\" (UniqueName: \"kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv\") pod \"payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:39.120604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.102083    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.120604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.102161    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:39.120604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.103191    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:39.142737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.142706    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpgwv\" (UniqueName: \"kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv\") pod \"payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:39.148978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.148953    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:39.203307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.203272    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.203475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.203339    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvzr\" (UniqueName: \"kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr\") pod \"payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.203994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.203975    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.226952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.226921    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvzr\" (UniqueName: \"kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr\") pod \"payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.237217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.237194    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:39.268748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.266014    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:39.314078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.314037    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:39.386512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.381991    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:39.589185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.589156    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"]
Oct 30 17:26:39.632738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.632610    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz" event={"ID":"b0d64771-f9aa-462c-bd01-2fa4d430d7df","Type":"ContainerStarted","Data":"8b15f9996493399845f98f3b6cc473709532f08ed39c774d7f3f8cd860995b27"}
Oct 30 17:26:39.785680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:39.785587    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"]
Oct 30 17:26:39.788600 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:39.788564    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82557d68_0eee_480b_960b_90087e394d52.slice/crio-40c6877daa9fc90dc5905e78118d25dee2431cdc9686baa4befb2aa43fad6492 WatchSource:0}: Error finding container 40c6877daa9fc90dc5905e78118d25dee2431cdc9686baa4befb2aa43fad6492: Status 404 returned error can't find the container with id 40c6877daa9fc90dc5905e78118d25dee2431cdc9686baa4befb2aa43fad6492
Oct 30 17:26:40.640356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.640318    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz" event={"ID":"b0d64771-f9aa-462c-bd01-2fa4d430d7df","Type":"ContainerStarted","Data":"381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9"}
Oct 30 17:26:40.640893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.640857    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:40.642336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.642313    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg" event={"ID":"82557d68-0eee-480b-960b-90087e394d52","Type":"ContainerStarted","Data":"b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263"}
Oct 30 17:26:40.642425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.642351    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg" event={"ID":"82557d68-0eee-480b-960b-90087e394d52","Type":"ContainerStarted","Data":"40c6877daa9fc90dc5905e78118d25dee2431cdc9686baa4befb2aa43fad6492"}
Oct 30 17:26:40.642768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.642754    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:40.732230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.732183    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz" podStartSLOduration=2.73216767 podStartE2EDuration="2.73216767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:40.690958352 +0000 UTC m=+806.453002168" watchObservedRunningTime="2026-10-30 17:26:40.73216767 +0000 UTC m=+806.494211555"
Oct 30 17:26:40.732758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:40.732729    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg" podStartSLOduration=2.732722355 podStartE2EDuration="2.732722355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:40.731556565 +0000 UTC m=+806.493600380" watchObservedRunningTime="2026-10-30 17:26:40.732722355 +0000 UTC m=+806.494766168"
Oct 30 17:26:41.299457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:41.299421    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:42.598067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:42.598036    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 17:26:42.600090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:42.600063    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 17:26:43.469666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:43.469072    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:43.593300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:43.593173    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:43.609670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:43.607376    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 17:26:43.609670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:43.608554    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 17:26:44.608828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:44.608793    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 17:26:45.249563 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:26:45.249494    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:26:45.895689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:45.895142    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:46.899670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:46.899459    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:48.069715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.069507    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"]
Oct 30 17:26:48.075682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.074535    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.088611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.088528    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"]
Oct 30 17:26:48.100349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.099541    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s8xph\""
Oct 30 17:26:48.100349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.100254    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:26:48.107310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.107195    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:26:48.195256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.195025    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-8d946d-serving-cert\""
Oct 30 17:26:48.195256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.195214    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 17:26:48.202707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.197459    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.202707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.197585    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.202707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.197698    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.202707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.197742    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts226\" (UniqueName: \"kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.300710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.299178    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.300710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.299248    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts226\" (UniqueName: \"kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.300710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.299294    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.300710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.299415    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.300710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.300265    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.306977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.306917    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.308427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.308378    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.326523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.326436    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts226\" (UniqueName: \"kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226\") pod \"maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.445393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.445368    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:48.462686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.462634    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:48.775032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:48.774565    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"]
Oct 30 17:26:48.776857 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:48.776816    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd401a478_02e0_4481_8e59_75a3d5c4caec.slice/crio-5d1cc77a15da59894c9316c438d87374c3e080c90f81f24f3ca0d530171d8d4c WatchSource:0}: Error finding container 5d1cc77a15da59894c9316c438d87374c3e080c90f81f24f3ca0d530171d8d4c: Status 404 returned error can't find the container with id 5d1cc77a15da59894c9316c438d87374c3e080c90f81f24f3ca0d530171d8d4c
Oct 30 17:26:49.779718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.779618    5534 generic.go:356] "Generic (PLEG): container finished" podID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerID="a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5" exitCode=0
Oct 30 17:26:49.780309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.779734    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" event={"ID":"c688455a-36a2-4f26-9ccd-20be9f3af129","Type":"ContainerDied","Data":"a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"}
Oct 30 17:26:49.780790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.780770    5534 scope.go:122] "RemoveContainer" containerID="a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"
Oct 30 17:26:49.783131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.782565    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:49.783131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.782702    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:49.783131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.782754    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:49.784437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:49.784410    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42" event={"ID":"d401a478-02e0-4481-8e59-75a3d5c4caec","Type":"ContainerStarted","Data":"5d1cc77a15da59894c9316c438d87374c3e080c90f81f24f3ca0d530171d8d4c"}
Oct 30 17:26:50.408396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:50.408357    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:50.543332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:50.543297    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:50.689744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:50.668178    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:50.795667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:50.795153    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:50.810141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:50.810105    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" event={"ID":"c688455a-36a2-4f26-9ccd-20be9f3af129","Type":"ContainerStarted","Data":"222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f"}
Oct 30 17:26:51.209313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.209072    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:51.210516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.210471    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 17:26:51.210608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.210523    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 17:26:51.287979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.287811    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:51.410045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.409972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:51.528261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.528224    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:51.650311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.650273    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 17:26:51.819460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.819329    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42" event={"ID":"d401a478-02e0-4481-8e59-75a3d5c4caec","Type":"ContainerStarted","Data":"b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb"}
Oct 30 17:26:51.819888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.819577    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:51.989668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:51.979455    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:52.110009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.109874    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:52.209412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.209078    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 17:26:52.209412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.209128    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 17:26:52.226540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.226384    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:52.737528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.735849    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 17:26:52.763439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.763064    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:52.786669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.778830    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42" podStartSLOduration=2.407518291 podStartE2EDuration="4.778811134s" totalImagesPullingTime="2.371292843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:51.867085747 +0000 UTC m=+817.629129561" watchObservedRunningTime="2026-10-30 17:26:52.778811134 +0000 UTC m=+818.540854948"
Oct 30 17:26:52.829384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.829352    5534 generic.go:356] "Generic (PLEG): container finished" podID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerID="e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd" exitCode=0
Oct 30 17:26:52.830559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.830532    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" event={"ID":"62cfe7a4-8d40-41b6-a066-a6619d20138a","Type":"ContainerDied","Data":"e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"}
Oct 30 17:26:52.831249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.831229    5534 scope.go:122] "RemoveContainer" containerID="e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"
Oct 30 17:26:52.833556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.833532    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:52.833760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.833742    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:52.833864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:52.833852    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:53.209164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.209122    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 17:26:53.209371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.209190    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 17:26:53.388679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.364903    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"]
Oct 30 17:26:53.400136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.396251    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"]
Oct 30 17:26:53.400136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.396374    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.421922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.421277    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"default-dockercfg-hnr9h\""
Oct 30 17:26:53.421922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.421345    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"kube-root-ca.crt\""
Oct 30 17:26:53.421922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.421792    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"simulator-w5-db98b0-kserve-self-signed-certs\""
Oct 30 17:26:53.422128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.422070    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"openshift-service-ca.crt\""
Oct 30 17:26:53.425000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.423990    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:53.515834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.515269    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.515834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.515326    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm76j\" (UniqueName: \"kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.515834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.515361    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.515834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.515408    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.515834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.515482    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.515834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.515544    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.629443    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.629519    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.629600    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.629684    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.629791    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.629817    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm76j\" (UniqueName: \"kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.637108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.631699    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.637108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.631956    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.637499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.637463    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.640303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.640278    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.640606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.640565    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.658665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.653553    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm76j\" (UniqueName: \"kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j\") pod \"simulator-w5-db98b0-kserve-85ffc4b59-p2zwq\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") " pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.727158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.727124    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:53.842965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.841073    5534 generic.go:356] "Generic (PLEG): container finished" podID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerID="2a5e06a4089631dff13a86b92f407ed5790e6cf7a085c4dcc7b69e64b5af5ade" exitCode=0
Oct 30 17:26:53.842965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.841181    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" event={"ID":"683cd04d-90d2-4f93-b86d-c72d0f33bf13","Type":"ContainerDied","Data":"2a5e06a4089631dff13a86b92f407ed5790e6cf7a085c4dcc7b69e64b5af5ade"}
Oct 30 17:26:53.842965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.842201    5534 scope.go:122] "RemoveContainer" containerID="2a5e06a4089631dff13a86b92f407ed5790e6cf7a085c4dcc7b69e64b5af5ade"
Oct 30 17:26:53.844939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.844401    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:53.844939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.844498    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:53.844939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.844548    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:26:53.847795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:53.847771    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" event={"ID":"62cfe7a4-8d40-41b6-a066-a6619d20138a","Type":"ContainerStarted","Data":"29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd"}
Oct 30 17:26:54.130237 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:26:54.130089    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcdefec2_1662_4bc5_b11b_308df189a01c.slice/crio-837d8d950395292133e234700a88f7c8d9011bacf8436b86c9ce74f618fc2cc7 WatchSource:0}: Error finding container 837d8d950395292133e234700a88f7c8d9011bacf8436b86c9ce74f618fc2cc7: Status 404 returned error can't find the container with id 837d8d950395292133e234700a88f7c8d9011bacf8436b86c9ce74f618fc2cc7
Oct 30 17:26:54.130424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.130390    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"]
Oct 30 17:26:54.227298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.227251    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 17:26:54.227444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.227312    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 17:26:54.519509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.519280    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:54.519898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.519876    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 17:26:54.520061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.520038    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 17:26:54.538129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.538108    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:54.538325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.538293    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:54.859668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.858744    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" event={"ID":"bcdefec2-1662-4bc5-b11b-308df189a01c","Type":"ContainerStarted","Data":"e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2"}
Oct 30 17:26:54.859668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.858801    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" event={"ID":"bcdefec2-1662-4bc5-b11b-308df189a01c","Type":"ContainerStarted","Data":"837d8d950395292133e234700a88f7c8d9011bacf8436b86c9ce74f618fc2cc7"}
Oct 30 17:26:54.863665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.863564    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" event={"ID":"683cd04d-90d2-4f93-b86d-c72d0f33bf13","Type":"ContainerStarted","Data":"810aa5e39d79d2e13754904c347f17694bee04c8a9c693cea0d04841912b3e9d"}
Oct 30 17:26:54.938259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:54.937799    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podStartSLOduration=1.937779437 podStartE2EDuration="1.937779437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:54.920172234 +0000 UTC m=+820.682216059" watchObservedRunningTime="2026-10-30 17:26:54.937779437 +0000 UTC m=+820.699823254"
Oct 30 17:26:55.004781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.004745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:55.209728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.209082    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 17:26:55.209728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.209145    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 17:26:55.511165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.511068    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 17:26:55.511474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.511436    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 17:26:55.542441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.542411    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:55.727837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.727796    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:26:55.781531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.781061    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 17:26:55.781531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.781125    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 17:26:55.781531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:55.781407    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:26:56.423201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.423170    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:56.424663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.424627    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 17:26:56.494132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.494073    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:56.511021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.510983    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 17:26:56.511293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.511249    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 17:26:56.617672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.617596    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:56.781728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.781079    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 17:26:56.781728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:56.781152    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 17:26:57.150128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.150041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:57.280669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.279133    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:57.513344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.512882    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 17:26:57.513344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.512941    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 17:26:57.513999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.513471    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:57.658325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.658279    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:26:57.785723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.782028    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 17:26:57.785723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.782095    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 17:26:57.836909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:57.836882    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 17:26:58.513665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:58.513036    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 17:26:58.513665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:58.513103    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 17:26:58.783730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:58.780067    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 17:26:58.783730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:58.780131    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 17:26:59.516110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:59.516081    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:59.517109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:59.517083    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 17:26:59.778188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:59.778080    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 17:26:59.778188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:59.778145    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 17:26:59.778683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:26:59.778609    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:00.926677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:00.924119    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:27:00.937678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:00.935863    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:27:01.577619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:01.577582    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:01.693217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:01.693165    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:01.816273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:01.816238    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:02.073546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:02.073505    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:02.606658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:02.606615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:02.755070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:02.755028    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:02.893733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:02.893625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:03.645879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:03.645845    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:03.728533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:03.728495    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:27:03.749175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:03.749141    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:27:04.021065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:04.020970    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:27:05.399498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:05.399455    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:06.787667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.787596    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"]
Oct 30 17:27:06.795409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.795382    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.816612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.816575    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"]
Oct 30 17:27:06.826353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.826323    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"simulator-w1-455086-kserve-self-signed-certs\""
Oct 30 17:27:06.826607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.826392    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"default-dockercfg-dwx7b\""
Oct 30 17:27:06.826607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.826416    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"kube-root-ca.crt\""
Oct 30 17:27:06.829213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.829189    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"openshift-service-ca.crt\""
Oct 30 17:27:06.857725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.857684    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.857725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.857738    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.858088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.857784    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6b6\" (UniqueName: \"kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.858088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.857850    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.858088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.857952    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.858088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.858056    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.873897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.873868    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:06.959354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959321    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959364    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959412    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6b6\" (UniqueName: \"kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959469    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959497    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959540    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959877    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.959965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.959915    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.960194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.960171    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.963680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.963170    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.963680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.963322    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:06.984216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:06.984182    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6b6\" (UniqueName: \"kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6\") pod \"simulator-w1-455086-kserve-db8fcdbfb-xw8j5\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") " pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:07.112514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:07.112480    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:07.316637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:07.316605    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:07.420378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:07.420345    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"]
Oct 30 17:27:07.422305 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:27:07.422273    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod526a3a13_0745_46f4_a20b_44ee8bceb95b.slice/crio-8da3f1e7305cc9833d5538fc9b36277185a24e94d05d569eef7e932fe289a304 WatchSource:0}: Error finding container 8da3f1e7305cc9833d5538fc9b36277185a24e94d05d569eef7e932fe289a304: Status 404 returned error can't find the container with id 8da3f1e7305cc9833d5538fc9b36277185a24e94d05d569eef7e932fe289a304
Oct 30 17:27:08.033983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:08.033948    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" event={"ID":"526a3a13-0745-46f4-a20b-44ee8bceb95b","Type":"ContainerStarted","Data":"c96d8aa4bd95fc3a9d1f5b1cab0b45b940d25ab77cb756ead43517eb7f184dc5"}
Oct 30 17:27:08.034469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:08.033993    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" event={"ID":"526a3a13-0745-46f4-a20b-44ee8bceb95b","Type":"ContainerStarted","Data":"8da3f1e7305cc9833d5538fc9b36277185a24e94d05d569eef7e932fe289a304"}
Oct 30 17:27:08.077630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:08.077571    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podStartSLOduration=2.077552406 podStartE2EDuration="2.077552406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:08.075905843 +0000 UTC m=+833.837949656" watchObservedRunningTime="2026-10-30 17:27:08.077552406 +0000 UTC m=+833.839596220"
Oct 30 17:27:08.113533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:08.113495    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:09.233262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:09.233221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:10.296004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.295966    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"]
Oct 30 17:27:10.299624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.299602    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.314504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.314482    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"]
Oct 30 17:27:10.327128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.327106    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"simulator-w3-306aed-kserve-self-signed-certs\""
Oct 30 17:27:10.327259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.327142    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"kube-root-ca.crt\""
Oct 30 17:27:10.327259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.327141    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"default-dockercfg-4sfjv\""
Oct 30 17:27:10.327343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.327288    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"openshift-service-ca.crt\""
Oct 30 17:27:10.349080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.349059    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:10.410574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.410531    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.410762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.410579    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.410839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.410813    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzg7\" (UniqueName: \"kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.411007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.410985    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.411050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.411034    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.411149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.411130    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.511763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.511730    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.511962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.511782    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.511962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.511828    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.511962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.511859    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.511962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.511891    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.512174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.512004    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzg7\" (UniqueName: \"kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.512254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.512226    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.512368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.512343    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.512459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.512440    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.514307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.514285    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.514445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.514427    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.541470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.538702    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzg7\" (UniqueName: \"kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7\") pod \"simulator-w3-306aed-kserve-55d9769d55-l9zgj\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") " pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.612943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.612907    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:10.839204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.839157    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:10.927279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:10.927254    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"]
Oct 30 17:27:10.929408 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:27:10.929366    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0519c04_2d73_4f8f_bf20_a88af61fbbf4.slice/crio-daf17196dd7930185fb66e37a72d8e3112deab0ca799d46108216c87b6bb3263 WatchSource:0}: Error finding container daf17196dd7930185fb66e37a72d8e3112deab0ca799d46108216c87b6bb3263: Status 404 returned error can't find the container with id daf17196dd7930185fb66e37a72d8e3112deab0ca799d46108216c87b6bb3263
Oct 30 17:27:11.052696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:11.052656    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" event={"ID":"b0519c04-2d73-4f8f-bf20-a88af61fbbf4","Type":"ContainerStarted","Data":"e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c"}
Oct 30 17:27:11.052886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:11.052713    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" event={"ID":"b0519c04-2d73-4f8f-bf20-a88af61fbbf4","Type":"ContainerStarted","Data":"daf17196dd7930185fb66e37a72d8e3112deab0ca799d46108216c87b6bb3263"}
Oct 30 17:27:11.095309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:11.095241    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podStartSLOduration=1.095222852 podStartE2EDuration="1.095222852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:11.093265378 +0000 UTC m=+836.855309193" watchObservedRunningTime="2026-10-30 17:27:11.095222852 +0000 UTC m=+836.857266659"
Oct 30 17:27:11.333793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:11.333693    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:11.613662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:11.613607    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:12.138774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:12.138739    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:12.259792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:12.259753    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:12.397821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:12.397732    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:12.843943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:12.843910    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:12.967587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:12.964578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:14.570835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:14.570802    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:15.245121 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:27:15.245069    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:27:16.286128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:16.286092    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:17.112709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:17.112673    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:17.126932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:17.126903    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:18.109017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:18.108983    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:27:18.208919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:18.208879    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:20.015425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:20.015394    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:20.613874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:20.613834    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:20.626927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:20.626893    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:21.046673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:21.046558    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:21.125447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:21.125421    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:27:21.713071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:21.713038    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:22.281971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:22.281935    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:22.914420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:22.914386    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:23.027247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:23.027213    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:23.149012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:23.148975    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:23.408074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:23.408042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:25.206574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:25.206541    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:26.969899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:26.969855    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:28.547906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:28.547868    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:28.708772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:28.708734    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:28.868853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:28.868825    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:30.425709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:30.425666    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:32.090025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.089990    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:32.503573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.503486    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"]
Oct 30 17:27:32.508241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.508210    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.540432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.540398    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"premium-w5-db98b0-kserve-self-signed-certs\""
Oct 30 17:27:32.543946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.543917    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"]
Oct 30 17:27:32.570329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.570287    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.570517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.570352    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.570517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.570453    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.570691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.570559    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxgv\" (UniqueName: \"kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.570691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.570617    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.570815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.570726    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.589222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.589187    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:32.672407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.672370    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.672592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.672422    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.672592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.672468    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.672592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.672509    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxgv\" (UniqueName: \"kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.672592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.672563    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.672826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.672622    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.673157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.673130    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.673356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.673180    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.673356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.673335    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.675471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.675443    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.675616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.675568    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.706669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.706331    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxgv\" (UniqueName: \"kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv\") pod \"premium-w5-db98b0-kserve-7dd8c7c454-8m729\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") " pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:32.823282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:32.823188    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:33.140128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:33.140103    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"]
Oct 30 17:27:33.142360 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:27:33.142327    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b804fac_5e8d_4b4f_a194_d17214118b8b.slice/crio-133720eb9b1a05ca2f88e5f8003037224a6479af6aa5693b9eb7aa36b43e7be2 WatchSource:0}: Error finding container 133720eb9b1a05ca2f88e5f8003037224a6479af6aa5693b9eb7aa36b43e7be2: Status 404 returned error can't find the container with id 133720eb9b1a05ca2f88e5f8003037224a6479af6aa5693b9eb7aa36b43e7be2
Oct 30 17:27:33.186065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:33.186018    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" event={"ID":"8b804fac-5e8d-4b4f-a194-d17214118b8b","Type":"ContainerStarted","Data":"133720eb9b1a05ca2f88e5f8003037224a6479af6aa5693b9eb7aa36b43e7be2"}
Oct 30 17:27:34.008539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:34.008502    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:34.192803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:34.192764    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" event={"ID":"8b804fac-5e8d-4b4f-a194-d17214118b8b","Type":"ContainerStarted","Data":"6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83"}
Oct 30 17:27:34.234062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:34.234012    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podStartSLOduration=2.233998143 podStartE2EDuration="2.233998143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:34.233378112 +0000 UTC m=+859.995421940" watchObservedRunningTime="2026-10-30 17:27:34.233998143 +0000 UTC m=+859.996041967"
Oct 30 17:27:34.824265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:34.824232    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:35.697939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:35.697901    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:36.271331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:36.271296    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:37.399466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:37.399433    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:39.220911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:39.220869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:40.932351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:40.932308    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:42.639547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:42.639510    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:42.777712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:42.777671    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:42.823728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:42.823690    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:42.836920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:42.836897    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:43.259588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:43.259561    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:27:43.603544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:43.603516    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:44.421287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:44.421257    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:45.243568 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:27:45.243520    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:27:46.101960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:46.101927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:48.002022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:48.001898    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:49.612950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:49.612915    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:50.859142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:50.859109    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:51.347666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:51.347611    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:52.934275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:52.934241    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:53.202267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:53.202189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:54.875774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:54.875737    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:55.738135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.738100    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"]
Oct 30 17:27:55.744820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.744788    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.784079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.784032    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"]
Oct 30 17:27:55.784280 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.784227    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"default-dockercfg-hdh9p\""
Oct 30 17:27:55.784360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.784344    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"openshift-service-ca.crt\""
Oct 30 17:27:55.784447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.784434    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"kube-root-ca.crt\""
Oct 30 17:27:55.784512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.784493    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"premium-w6-8d946d-kserve-self-signed-certs\""
Oct 30 17:27:55.805857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.805820    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:55.844421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.844383    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.844598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.844529    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rzv\" (UniqueName: \"kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.844689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.844612    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.844689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.844667    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.844790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.844723    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.844825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.844805    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.946587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.946550    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rzv\" (UniqueName: \"kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.947040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.946611    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.947040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.946683    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.947040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.946736    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.947040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.946788    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.947040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.946848    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.947495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.947468    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.948313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.947553    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.948474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.948252    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.951966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.951943    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.954189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.954165    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:55.982668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:55.971349    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rzv\" (UniqueName: \"kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv\") pod \"premium-w6-8d946d-kserve-5c79445f8f-49sks\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") " pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:56.076387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:56.076293    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:56.416095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:56.416067    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"]
Oct 30 17:27:56.417940 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:27:56.417905    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46fcbe49_cbaf_4844_9866_efb0a540e624.slice/crio-991b33f7b1f15da33d66deedbe687cab108b05e32825ecc13ec1fedd23284b86 WatchSource:0}: Error finding container 991b33f7b1f15da33d66deedbe687cab108b05e32825ecc13ec1fedd23284b86: Status 404 returned error can't find the container with id 991b33f7b1f15da33d66deedbe687cab108b05e32825ecc13ec1fedd23284b86
Oct 30 17:27:56.735668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:56.735533    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:57.326794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:57.326745    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" event={"ID":"46fcbe49-cbaf-4844-9866-efb0a540e624","Type":"ContainerStarted","Data":"bca7bf24103a2c7f91f1f84336205d6f9652b834723c4f796c075b43035d8c3d"}
Oct 30 17:27:57.327209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:57.326809    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" event={"ID":"46fcbe49-cbaf-4844-9866-efb0a540e624","Type":"ContainerStarted","Data":"991b33f7b1f15da33d66deedbe687cab108b05e32825ecc13ec1fedd23284b86"}
Oct 30 17:27:57.404655 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:57.404577    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podStartSLOduration=2.404563156 podStartE2EDuration="2.404563156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:57.402225123 +0000 UTC m=+883.164268938" watchObservedRunningTime="2026-10-30 17:27:57.404563156 +0000 UTC m=+883.166606968"
Oct 30 17:27:58.077116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:58.077074    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:27:58.385173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:58.385143    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:27:58.758042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:27:58.757954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:00.020174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:00.020132    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:01.628960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:01.628924    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:03.420746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:03.420708    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:05.220266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.220228    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"]
Oct 30 17:28:05.224961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.224932    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.225997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.225960    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:05.242935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.242903    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"]
Oct 30 17:28:05.252667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.251765    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"premium-w2-506124-kserve-self-signed-certs\""
Oct 30 17:28:05.252667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.251805    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"default-dockercfg-89cmx\""
Oct 30 17:28:05.252910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.252767    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"kube-root-ca.crt\""
Oct 30 17:28:05.252910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.252819    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"openshift-service-ca.crt\""
Oct 30 17:28:05.343665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.343425    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:05.367664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.367546    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.367664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.367615    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.372800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.371800    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.375716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.375688    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.375939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.375924    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.376119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.376088    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrbd\" (UniqueName: \"kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.481726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.481407    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.481726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.481469    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.481726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.481534    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.481726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.481593    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.481726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.481622    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.481726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.481693    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrbd\" (UniqueName: \"kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.483492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.483464    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.483635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.483538    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.484138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.484106    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.509410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.509384    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.509654 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.509547    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrbd\" (UniqueName: \"kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.532037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.531969    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm\") pod \"premium-w2-506124-kserve-6cc4754cc4-rgv9t\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") " pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:05.543139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:05.542873    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:06.001261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.001227    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"]
Oct 30 17:28:06.003208 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:06.003180    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7bdf7e9_e0e6_4865_927e_418b7a7bc236.slice/crio-f4a70d98285bb716c8bfc5430036b9c20f9f4299fbc673a00ac0fc48dc040b6d WatchSource:0}: Error finding container f4a70d98285bb716c8bfc5430036b9c20f9f4299fbc673a00ac0fc48dc040b6d: Status 404 returned error can't find the container with id f4a70d98285bb716c8bfc5430036b9c20f9f4299fbc673a00ac0fc48dc040b6d
Oct 30 17:28:06.076678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.076609    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:28:06.100678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.100375    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:28:06.389734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.389692    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" event={"ID":"e7bdf7e9-e0e6-4865-927e-418b7a7bc236","Type":"ContainerStarted","Data":"145cb8965faf903610e39a27063de1f1f744f8765e00e9a8d78eba36e46f009f"}
Oct 30 17:28:06.390256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.389750    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" event={"ID":"e7bdf7e9-e0e6-4865-927e-418b7a7bc236","Type":"ContainerStarted","Data":"f4a70d98285bb716c8bfc5430036b9c20f9f4299fbc673a00ac0fc48dc040b6d"}
Oct 30 17:28:06.402289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.402263    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:28:06.512082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.512022    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podStartSLOduration=1.5120028890000001 podStartE2EDuration="1.512002889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:06.456335911 +0000 UTC m=+892.218379728" watchObservedRunningTime="2026-10-30 17:28:06.512002889 +0000 UTC m=+892.274046703"
Oct 30 17:28:06.544855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:06.544219    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:07.323056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.322997    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:07.346948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.346912    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"]
Oct 30 17:28:07.350982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.350960    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.366168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.366140    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-db98b0-openshift-default-dockercfg-qvj6k\""
Oct 30 17:28:07.381725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.381689    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"]
Oct 30 17:28:07.414870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.414839    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.414870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.414872    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.414897    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.415054    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.415199    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.415256    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.415328    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.415372    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvwq\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.415485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.415462    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516185    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516226    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516250    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvwq\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516283    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516390    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516418    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516450    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516504    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516701    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.516904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516751    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.517112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.516702    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.517567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.517534    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.517755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.517733    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.518050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.517997    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.519604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.519579    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.519715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.519634    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.545755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.545677    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.546076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.546056    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvwq\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:07.669246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:07.669208    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:08.014961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.014779    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"]
Oct 30 17:28:08.018032 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:08.018003    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f21b507_00b3_4988_b1b6_1dabe8f90ca9.slice/crio-90252ba551fc6977dee98f9761509dcdb2322423c58e2d1ff135eb6c04d233fb WatchSource:0}: Error finding container 90252ba551fc6977dee98f9761509dcdb2322423c58e2d1ff135eb6c04d233fb: Status 404 returned error can't find the container with id 90252ba551fc6977dee98f9761509dcdb2322423c58e2d1ff135eb6c04d233fb
Oct 30 17:28:08.020999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.020959    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:28:08.021083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.021052    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:28:08.021147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.021105    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:28:08.409052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.409012    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" event={"ID":"3f21b507-00b3-4988-b1b6-1dabe8f90ca9","Type":"ContainerStarted","Data":"a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce"}
Oct 30 17:28:08.409271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.409086    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" event={"ID":"3f21b507-00b3-4988-b1b6-1dabe8f90ca9","Type":"ContainerStarted","Data":"90252ba551fc6977dee98f9761509dcdb2322423c58e2d1ff135eb6c04d233fb"}
Oct 30 17:28:08.460283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.460216    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podStartSLOduration=1.460196382 podStartE2EDuration="1.460196382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:08.457126713 +0000 UTC m=+894.219170528" watchObservedRunningTime="2026-10-30 17:28:08.460196382 +0000 UTC m=+894.222240202"
Oct 30 17:28:08.673731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:08.672739    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:09.085279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:09.085154    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:09.674668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:09.673221    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:28:09.674668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:09.673300    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:28:10.676668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:10.676243    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:28:10.676668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:10.676303    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:28:10.831116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:10.831081    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:11.676670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:11.676026    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:28:11.676670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:11.676096    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:28:12.669667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:12.668139    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:12.674666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:12.673749    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:28:12.674666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:12.673803    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:28:13.672669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:13.672146    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:28:13.672669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:13.672224    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:28:14.170240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.170208    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"]
Oct 30 17:28:14.174105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.174086    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.191873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.191666    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:28:14.191993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.191965    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:28:14.194966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.194938    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"]
Oct 30 17:28:14.229368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.229327    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pth9k\" (UniqueName: \"kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.229551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.229517    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.229673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.229635    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.330909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.330868    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.331065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.330940    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.331065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.331005    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pth9k\" (UniqueName: \"kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.331816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.331778    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.344328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.344306    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:28:14.354208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.354166    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.354330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.354312    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pth9k\" (UniqueName: \"kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k\") pod \"maas-api-revoke-sub-46935f91-kh28k\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.504524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.504444    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:28:14.511540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.511514    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:14.517599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.517575    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:14.845240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.845213    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"]
Oct 30 17:28:14.847027 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:14.847001    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648c76ad_7072_49c4_95ee_3ff8c79738dc.slice/crio-441ce1e8febed4f7235f6566a82abfa05d1db65c65b6064a711c81686ef00c84 WatchSource:0}: Error finding container 441ce1e8febed4f7235f6566a82abfa05d1db65c65b6064a711c81686ef00c84: Status 404 returned error can't find the container with id 441ce1e8febed4f7235f6566a82abfa05d1db65c65b6064a711c81686ef00c84
Oct 30 17:28:14.849526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.849510    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:28:14.919443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.919411    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:14.919979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.919952    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:14.921381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:14.921358    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:28:15.244224 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:28:15.244139    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:28:15.451428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:15.451392    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k" event={"ID":"648c76ad-7072-49c4-95ee-3ff8c79738dc","Type":"ContainerStarted","Data":"441ce1e8febed4f7235f6566a82abfa05d1db65c65b6064a711c81686ef00c84"}
Oct 30 17:28:15.545346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:15.544465    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:15.563616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:15.563582    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:16.413680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:16.413079    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:16.472391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:16.472363    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:28:18.304505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:18.304417    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:18.490836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:18.490765    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k" event={"ID":"648c76ad-7072-49c4-95ee-3ff8c79738dc","Type":"ContainerStarted","Data":"eae3a2c14ec7c12eb230597b283c2a1e8457c38743e64522378b5f180f80b0a2"}
Oct 30 17:28:18.532661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:18.532594    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k" podStartSLOduration=1.6739186209999999 podStartE2EDuration="4.532577739s" totalImagesPullingTime="2.858659118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:18.530832236 +0000 UTC m=+904.292876054" watchObservedRunningTime="2026-10-30 17:28:18.532577739 +0000 UTC m=+904.294621549"
Oct 30 17:28:19.498948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:19.498909    5534 generic.go:356] "Generic (PLEG): container finished" podID="648c76ad-7072-49c4-95ee-3ff8c79738dc" containerID="eae3a2c14ec7c12eb230597b283c2a1e8457c38743e64522378b5f180f80b0a2" exitCode=0
Oct 30 17:28:19.498948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:19.499007    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k" event={"ID":"648c76ad-7072-49c4-95ee-3ff8c79738dc","Type":"ContainerDied","Data":"eae3a2c14ec7c12eb230597b283c2a1e8457c38743e64522378b5f180f80b0a2"}
Oct 30 17:28:20.035475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.035442    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:20.654671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.654620    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:20.747627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.747597    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca\") pod \"648c76ad-7072-49c4-95ee-3ff8c79738dc\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") "
Oct 30 17:28:20.747801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.747636    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k\" (UniqueName: \"kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k\") pod \"648c76ad-7072-49c4-95ee-3ff8c79738dc\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") "
Oct 30 17:28:20.747801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.747790    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca\") pod \"648c76ad-7072-49c4-95ee-3ff8c79738dc\" (UID: \"648c76ad-7072-49c4-95ee-3ff8c79738dc\") "
Oct 30 17:28:20.748223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.748189    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca" pod "648c76ad-7072-49c4-95ee-3ff8c79738dc" (UID: "648c76ad-7072-49c4-95ee-3ff8c79738dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:20.750660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.750601    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca" pod "648c76ad-7072-49c4-95ee-3ff8c79738dc" (UID: "648c76ad-7072-49c4-95ee-3ff8c79738dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:20.750660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.750613    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k" pod "648c76ad-7072-49c4-95ee-3ff8c79738dc" (UID: "648c76ad-7072-49c4-95ee-3ff8c79738dc"). InnerVolumeSpecName "kube-api-access-pth9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:20.848897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.848862    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:20.848897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.848893    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648c76ad-7072-49c4-95ee-3ff8c79738dc-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:20.848897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:20.848902    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pth9k\" (UniqueName: \"kubernetes.io/projected/648c76ad-7072-49c4-95ee-3ff8c79738dc-kube-api-access-pth9k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:21.512510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:21.512475    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k" event={"ID":"648c76ad-7072-49c4-95ee-3ff8c79738dc","Type":"ContainerDied","Data":"441ce1e8febed4f7235f6566a82abfa05d1db65c65b6064a711c81686ef00c84"}
Oct 30 17:28:21.512735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:21.512535    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441ce1e8febed4f7235f6566a82abfa05d1db65c65b6064a711c81686ef00c84"
Oct 30 17:28:21.512735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:21.512599    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"
Oct 30 17:28:21.674679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:21.674615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:21.785923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:21.785830    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:23.425794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:23.425758    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:25.217550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:25.217465    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:26.909084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:26.909051    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:28.595989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:28.595947    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:28.713361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:28.713326    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:30.411158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:30.411108    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:32.316226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:32.316198    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:33.579256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.579224    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"]
Oct 30 17:28:33.579829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.579812    5534 cpu_manager.go:414] "containerMap: removing container" podUID="648c76ad-7072-49c4-95ee-3ff8c79738dc" containerName="revoke-keys"
Oct 30 17:28:33.579885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.579830    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648c76ad-7072-49c4-95ee-3ff8c79738dc" containerName="revoke-keys"
Oct 30 17:28:33.579947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.579936    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="648c76ad-7072-49c4-95ee-3ff8c79738dc" containerName="revoke-keys"
Oct 30 17:28:33.583484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.583465    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.600887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.600842    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:28:33.612949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.612917    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:28:33.616462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.616424    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"]
Oct 30 17:28:33.750980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.750950    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.751177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.751079    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.751177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.751114    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzs2b\" (UniqueName: \"kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.852891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.852840    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.853167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.853140    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzs2b\" (UniqueName: \"kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.853351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.853334    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.854077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.854052    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.855952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.855926    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.877278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.877238    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzs2b\" (UniqueName: \"kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b\") pod \"maas-api-revoke-sub-3d9811f2-b7q2t\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:33.899247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:33.899218    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:34.152284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:34.152204    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:34.220935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:34.220903    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"]
Oct 30 17:28:34.601390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:34.601348    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t" event={"ID":"6da9ebc5-7bef-4170-a28d-f948e2eba7a2","Type":"ContainerStarted","Data":"07cfee76effcff1d76ea5186836f2cd92b9b8b6fb693d5c92fad014fff028dc0"}
Oct 30 17:28:34.601788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:34.601416    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t" event={"ID":"6da9ebc5-7bef-4170-a28d-f948e2eba7a2","Type":"ContainerStarted","Data":"61722ef7b859f1b9f32d1940c2b3f6975f23e51f00450d1b3e59ef13e5bff0e2"}
Oct 30 17:28:34.641565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:34.641504    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t" podStartSLOduration=1.6414867499999999 podStartE2EDuration="1.64148675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:34.641403737 +0000 UTC m=+920.403447551" watchObservedRunningTime="2026-10-30 17:28:34.64148675 +0000 UTC m=+920.403530565"
Oct 30 17:28:35.609283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:35.609239    5534 generic.go:356] "Generic (PLEG): container finished" podID="6da9ebc5-7bef-4170-a28d-f948e2eba7a2" containerID="07cfee76effcff1d76ea5186836f2cd92b9b8b6fb693d5c92fad014fff028dc0" exitCode=0
Oct 30 17:28:35.609836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:35.609331    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t" event={"ID":"6da9ebc5-7bef-4170-a28d-f948e2eba7a2","Type":"ContainerDied","Data":"07cfee76effcff1d76ea5186836f2cd92b9b8b6fb693d5c92fad014fff028dc0"}
Oct 30 17:28:35.786583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:35.786524    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:36.773943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.773916    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:36.801669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.801127    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b\" (UniqueName: \"kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b\") pod \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") "
Oct 30 17:28:36.801669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.801187    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca\") pod \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") "
Oct 30 17:28:36.801669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.801317    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca\") pod \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\" (UID: \"6da9ebc5-7bef-4170-a28d-f948e2eba7a2\") "
Oct 30 17:28:36.804677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.802071    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca" pod "6da9ebc5-7bef-4170-a28d-f948e2eba7a2" (UID: "6da9ebc5-7bef-4170-a28d-f948e2eba7a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:36.804677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.804409    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca" pod "6da9ebc5-7bef-4170-a28d-f948e2eba7a2" (UID: "6da9ebc5-7bef-4170-a28d-f948e2eba7a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:36.804677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.804451    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b" pod "6da9ebc5-7bef-4170-a28d-f948e2eba7a2" (UID: "6da9ebc5-7bef-4170-a28d-f948e2eba7a2"). InnerVolumeSpecName "kube-api-access-fzs2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:36.903183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.903102    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzs2b\" (UniqueName: \"kubernetes.io/projected/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-kube-api-access-fzs2b\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:36.903183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.903129    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:36.903183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:36.903139    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da9ebc5-7bef-4170-a28d-f948e2eba7a2-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:37.490378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:37.490346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:37.621511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:37.621482    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"
Oct 30 17:28:37.621757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:37.621506    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t" event={"ID":"6da9ebc5-7bef-4170-a28d-f948e2eba7a2","Type":"ContainerDied","Data":"61722ef7b859f1b9f32d1940c2b3f6975f23e51f00450d1b3e59ef13e5bff0e2"}
Oct 30 17:28:37.621757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:37.621561    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61722ef7b859f1b9f32d1940c2b3f6975f23e51f00450d1b3e59ef13e5bff0e2"
Oct 30 17:28:39.432265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:39.432207    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:41.038032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:41.037998    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:42.751757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:42.751720    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:43.737001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:43.736962    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:44.463256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:44.463193    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:45.245010 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:28:45.244972    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:28:45.672667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.671267    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"]
Oct 30 17:28:45.672667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.672026    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6da9ebc5-7bef-4170-a28d-f948e2eba7a2" containerName="revoke-keys"
Oct 30 17:28:45.672667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.672044    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da9ebc5-7bef-4170-a28d-f948e2eba7a2" containerName="revoke-keys"
Oct 30 17:28:45.672667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.672273    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da9ebc5-7bef-4170-a28d-f948e2eba7a2" containerName="revoke-keys"
Oct 30 17:28:45.688666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.687817    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.703664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.700465    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"]
Oct 30 17:28:45.713689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.712203    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:28:45.713689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.712360    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:28:45.841673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.841398    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlzf\" (UniqueName: \"kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.841673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.841520    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.841673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.841558    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.943091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.942999    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlzf\" (UniqueName: \"kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.943413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.943395    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.943544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.943531    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.944451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.944429    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.966306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.965681    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.985574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.985543    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlzf\" (UniqueName: \"kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf\") pod \"maas-api-revoke-sub-ca9fcb39-pv5vl\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:45.996478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.987530    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"]
Oct 30 17:28:45.996478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:45.992311    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.003895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.003856    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"]
Oct 30 17:28:46.024120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.024089    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:46.143849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.143793    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:46.146392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.146175    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.146392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.146247    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7t9\" (UniqueName: \"kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.146392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.146270    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.247421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.247322    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.247421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.247395    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7t9\" (UniqueName: \"kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.247421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.247422    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.248183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.248148    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.250128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.250107    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.271255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.271218    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7t9\" (UniqueName: \"kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9\") pod \"maas-api-revoke-sub-3ff628fa-htp5v\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.321290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.321252    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:46.341844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.341823    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"]
Oct 30 17:28:46.344349 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:46.344317    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651dd041_bd27_41dc_b84d_569438d1b955.slice/crio-d00947cb8a81f1131ceb01b27b2c76c1f5e37e1c15568c3e855608765f7aa41b WatchSource:0}: Error finding container d00947cb8a81f1131ceb01b27b2c76c1f5e37e1c15568c3e855608765f7aa41b: Status 404 returned error can't find the container with id d00947cb8a81f1131ceb01b27b2c76c1f5e37e1c15568c3e855608765f7aa41b
Oct 30 17:28:46.631951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.631926    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"]
Oct 30 17:28:46.633939 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:46.633913    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e32c42_9eac_4e3b_a194_236431278de8.slice/crio-117c44ba92d00c1e5d1e34e2b021086b497a20b49126ce8f04df57a66396bba2 WatchSource:0}: Error finding container 117c44ba92d00c1e5d1e34e2b021086b497a20b49126ce8f04df57a66396bba2: Status 404 returned error can't find the container with id 117c44ba92d00c1e5d1e34e2b021086b497a20b49126ce8f04df57a66396bba2
Oct 30 17:28:46.681100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.681060    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl" event={"ID":"651dd041-bd27-41dc-b84d-569438d1b955","Type":"ContainerStarted","Data":"1749dabad17deeb3b68769e041611a8d5159d5357c2df483f056e0614d9b2c0f"}
Oct 30 17:28:46.681544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.681138    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl" event={"ID":"651dd041-bd27-41dc-b84d-569438d1b955","Type":"ContainerStarted","Data":"d00947cb8a81f1131ceb01b27b2c76c1f5e37e1c15568c3e855608765f7aa41b"}
Oct 30 17:28:46.682486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.682464    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v" event={"ID":"15e32c42-9eac-4e3b-a194-236431278de8","Type":"ContainerStarted","Data":"117c44ba92d00c1e5d1e34e2b021086b497a20b49126ce8f04df57a66396bba2"}
Oct 30 17:28:46.721950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:46.721894    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl" podStartSLOduration=1.721878286 podStartE2EDuration="1.721878286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:46.720184907 +0000 UTC m=+932.482228716" watchObservedRunningTime="2026-10-30 17:28:46.721878286 +0000 UTC m=+932.483922099"
Oct 30 17:28:47.691959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:47.691924    5534 generic.go:356] "Generic (PLEG): container finished" podID="651dd041-bd27-41dc-b84d-569438d1b955" containerID="1749dabad17deeb3b68769e041611a8d5159d5357c2df483f056e0614d9b2c0f" exitCode=0
Oct 30 17:28:47.692428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:47.692006    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl" event={"ID":"651dd041-bd27-41dc-b84d-569438d1b955","Type":"ContainerDied","Data":"1749dabad17deeb3b68769e041611a8d5159d5357c2df483f056e0614d9b2c0f"}
Oct 30 17:28:47.697071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:47.696831    5534 generic.go:356] "Generic (PLEG): container finished" podID="15e32c42-9eac-4e3b-a194-236431278de8" containerID="7dfdf68d71c30519595254eb191c40735ff6e1340a1484d2a1d20bdd7fddfedd" exitCode=0
Oct 30 17:28:47.697071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:47.696924    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v" event={"ID":"15e32c42-9eac-4e3b-a194-236431278de8","Type":"ContainerDied","Data":"7dfdf68d71c30519595254eb191c40735ff6e1340a1484d2a1d20bdd7fddfedd"}
Oct 30 17:28:47.927729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:47.927691    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:48.585063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.585024    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" podUID="cd29d125f430f9e0a8a7b7b6580a8dec" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 17:28:48.899223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.898841    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:48.935573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.935547    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:48.983918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.983537    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf\" (UniqueName: \"kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf\") pod \"651dd041-bd27-41dc-b84d-569438d1b955\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") "
Oct 30 17:28:48.984183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.983986    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca\") pod \"651dd041-bd27-41dc-b84d-569438d1b955\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") "
Oct 30 17:28:48.984183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.984133    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca\") pod \"651dd041-bd27-41dc-b84d-569438d1b955\" (UID: \"651dd041-bd27-41dc-b84d-569438d1b955\") "
Oct 30 17:28:48.984669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.984487    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca" pod "651dd041-bd27-41dc-b84d-569438d1b955" (UID: "651dd041-bd27-41dc-b84d-569438d1b955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:48.984775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.984690    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:49.000998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:48.998759    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf" pod "651dd041-bd27-41dc-b84d-569438d1b955" (UID: "651dd041-bd27-41dc-b84d-569438d1b955"). InnerVolumeSpecName "kube-api-access-ktlzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:49.000998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.000817    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca" pod "651dd041-bd27-41dc-b84d-569438d1b955" (UID: "651dd041-bd27-41dc-b84d-569438d1b955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:49.085987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.085954    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9\" (UniqueName: \"kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9\") pod \"15e32c42-9eac-4e3b-a194-236431278de8\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") "
Oct 30 17:28:49.086177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.086100    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca\") pod \"15e32c42-9eac-4e3b-a194-236431278de8\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") "
Oct 30 17:28:49.086177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.086167    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca\") pod \"15e32c42-9eac-4e3b-a194-236431278de8\" (UID: \"15e32c42-9eac-4e3b-a194-236431278de8\") "
Oct 30 17:28:49.086666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.086507    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dd041-bd27-41dc-b84d-569438d1b955-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:49.086666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.086528    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktlzf\" (UniqueName: \"kubernetes.io/projected/651dd041-bd27-41dc-b84d-569438d1b955-kube-api-access-ktlzf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:49.086666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.086609    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca" pod "15e32c42-9eac-4e3b-a194-236431278de8" (UID: "15e32c42-9eac-4e3b-a194-236431278de8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:49.088697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.088674    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9" pod "15e32c42-9eac-4e3b-a194-236431278de8" (UID: "15e32c42-9eac-4e3b-a194-236431278de8"). InnerVolumeSpecName "kube-api-access-rj7t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:49.089018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.088994    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca" pod "15e32c42-9eac-4e3b-a194-236431278de8" (UID: "15e32c42-9eac-4e3b-a194-236431278de8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:49.187830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.187744    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7t9\" (UniqueName: \"kubernetes.io/projected/15e32c42-9eac-4e3b-a194-236431278de8-kube-api-access-rj7t9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:49.187830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.187773    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:49.187830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.187783    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e32c42-9eac-4e3b-a194-236431278de8-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:49.711079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.711040    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"
Oct 30 17:28:49.711261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.711040    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl" event={"ID":"651dd041-bd27-41dc-b84d-569438d1b955","Type":"ContainerDied","Data":"d00947cb8a81f1131ceb01b27b2c76c1f5e37e1c15568c3e855608765f7aa41b"}
Oct 30 17:28:49.711261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.711188    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00947cb8a81f1131ceb01b27b2c76c1f5e37e1c15568c3e855608765f7aa41b"
Oct 30 17:28:49.712506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.712479    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"
Oct 30 17:28:49.712691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.712506    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v" event={"ID":"15e32c42-9eac-4e3b-a194-236431278de8","Type":"ContainerDied","Data":"117c44ba92d00c1e5d1e34e2b021086b497a20b49126ce8f04df57a66396bba2"}
Oct 30 17:28:49.712691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.712552    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117c44ba92d00c1e5d1e34e2b021086b497a20b49126ce8f04df57a66396bba2"
Oct 30 17:28:49.743180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:49.743145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:50.239795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.239765    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"]
Oct 30 17:28:50.240256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.240243    5534 cpu_manager.go:414] "containerMap: removing container" podUID="651dd041-bd27-41dc-b84d-569438d1b955" containerName="revoke-keys"
Oct 30 17:28:50.240304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.240257    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651dd041-bd27-41dc-b84d-569438d1b955" containerName="revoke-keys"
Oct 30 17:28:50.240304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.240276    5534 cpu_manager.go:414] "containerMap: removing container" podUID="15e32c42-9eac-4e3b-a194-236431278de8" containerName="revoke-keys"
Oct 30 17:28:50.240304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.240281    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e32c42-9eac-4e3b-a194-236431278de8" containerName="revoke-keys"
Oct 30 17:28:50.240408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.240370    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e32c42-9eac-4e3b-a194-236431278de8" containerName="revoke-keys"
Oct 30 17:28:50.240408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.240385    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="651dd041-bd27-41dc-b84d-569438d1b955" containerName="revoke-keys"
Oct 30 17:28:50.243929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.243912    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.263076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.263052    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:28:50.272322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.272277    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:28:50.275985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.275962    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"]
Oct 30 17:28:50.405140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.405106    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.405308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.405183    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gnj4\" (UniqueName: \"kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.405308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.405232    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.507020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.506946    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.507020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.506978    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gnj4\" (UniqueName: \"kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.507225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.507034    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.507613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.507590    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.509531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.509505    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.530225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.530203    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gnj4\" (UniqueName: \"kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4\") pod \"maas-api-revoke-sub-8a9df3f8-8q7lw\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.557257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.557221    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:50.865868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:50.865836    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"]
Oct 30 17:28:50.866309 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:50.866282    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042fc1c6_886d_4115_ac27_c200af8378e3.slice/crio-7a1461239a514cc92110f94daf9a0f3d420237ffca31a9b8046239238026560d WatchSource:0}: Error finding container 7a1461239a514cc92110f94daf9a0f3d420237ffca31a9b8046239238026560d: Status 404 returned error can't find the container with id 7a1461239a514cc92110f94daf9a0f3d420237ffca31a9b8046239238026560d
Oct 30 17:28:51.431574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:51.431471    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:51.725975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:51.725932    5534 generic.go:356] "Generic (PLEG): container finished" podID="042fc1c6-886d-4115-ac27-c200af8378e3" containerID="69e4716895f62ae4285f7fc2b1dfc2b6bcc527e83b9acc375659f74542642814" exitCode=0
Oct 30 17:28:51.726125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:51.726021    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw" event={"ID":"042fc1c6-886d-4115-ac27-c200af8378e3","Type":"ContainerDied","Data":"69e4716895f62ae4285f7fc2b1dfc2b6bcc527e83b9acc375659f74542642814"}
Oct 30 17:28:51.726125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:51.726117    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw" event={"ID":"042fc1c6-886d-4115-ac27-c200af8378e3","Type":"ContainerStarted","Data":"7a1461239a514cc92110f94daf9a0f3d420237ffca31a9b8046239238026560d"}
Oct 30 17:28:52.958720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:52.958690    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:53.046117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.045371    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4\" (UniqueName: \"kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4\") pod \"042fc1c6-886d-4115-ac27-c200af8378e3\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") "
Oct 30 17:28:53.046117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.045437    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca\") pod \"042fc1c6-886d-4115-ac27-c200af8378e3\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") "
Oct 30 17:28:53.046117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.045623    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca\") pod \"042fc1c6-886d-4115-ac27-c200af8378e3\" (UID: \"042fc1c6-886d-4115-ac27-c200af8378e3\") "
Oct 30 17:28:53.046117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.045996    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca" pod "042fc1c6-886d-4115-ac27-c200af8378e3" (UID: "042fc1c6-886d-4115-ac27-c200af8378e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:53.048239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.048207    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4" pod "042fc1c6-886d-4115-ac27-c200af8378e3" (UID: "042fc1c6-886d-4115-ac27-c200af8378e3"). InnerVolumeSpecName "kube-api-access-4gnj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:53.048581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.048543    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca" pod "042fc1c6-886d-4115-ac27-c200af8378e3" (UID: "042fc1c6-886d-4115-ac27-c200af8378e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:53.150661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.145553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:53.150661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.148470    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:53.150661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.148495    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gnj4\" (UniqueName: \"kubernetes.io/projected/042fc1c6-886d-4115-ac27-c200af8378e3-kube-api-access-4gnj4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:53.150661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.148511    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042fc1c6-886d-4115-ac27-c200af8378e3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:28:53.369666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.355486    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:53.783738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.783378    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw" event={"ID":"042fc1c6-886d-4115-ac27-c200af8378e3","Type":"ContainerDied","Data":"7a1461239a514cc92110f94daf9a0f3d420237ffca31a9b8046239238026560d"}
Oct 30 17:28:53.783738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.783545    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1461239a514cc92110f94daf9a0f3d420237ffca31a9b8046239238026560d"
Oct 30 17:28:53.783738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:53.783694    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"
Oct 30 17:28:55.591668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:55.587771    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:57.502965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:57.502934    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:58.036566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.036526    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"]
Oct 30 17:28:58.037110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.037092    5534 cpu_manager.go:414] "containerMap: removing container" podUID="042fc1c6-886d-4115-ac27-c200af8378e3" containerName="revoke-keys"
Oct 30 17:28:58.037110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.037109    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042fc1c6-886d-4115-ac27-c200af8378e3" containerName="revoke-keys"
Oct 30 17:28:58.037306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.037197    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="042fc1c6-886d-4115-ac27-c200af8378e3" containerName="revoke-keys"
Oct 30 17:28:58.041473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.041447    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.061916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.060517    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"]
Oct 30 17:28:58.068680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.067575    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:28:58.068680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.067773    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:28:58.155942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.155821    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.156189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.156013    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.156189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.156054    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qbp\" (UniqueName: \"kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.257676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.257609    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.257872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.257796    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.257872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.257837    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qbp\" (UniqueName: \"kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.258410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.258383    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.260337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.260315    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.281400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.281376    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qbp\" (UniqueName: \"kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp\") pod \"maas-api-revoke-sub-c0b56010-njlgq\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.369118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.369094    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:28:58.670165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.670132    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"]
Oct 30 17:28:58.671302 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:28:58.671273    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0044f242_b3e3_44d6_897b_c147c1ff6118.slice/crio-e50fae9a905adf4ddf0fd193bbe8ceddda94ae02c87e0b87b7826370c001e390 WatchSource:0}: Error finding container e50fae9a905adf4ddf0fd193bbe8ceddda94ae02c87e0b87b7826370c001e390: Status 404 returned error can't find the container with id e50fae9a905adf4ddf0fd193bbe8ceddda94ae02c87e0b87b7826370c001e390
Oct 30 17:28:58.842458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:58.842418    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq" event={"ID":"0044f242-b3e3-44d6-897b-c147c1ff6118","Type":"ContainerStarted","Data":"e50fae9a905adf4ddf0fd193bbe8ceddda94ae02c87e0b87b7826370c001e390"}
Oct 30 17:28:59.194074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.194037    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:28:59.849751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.849712    5534 generic.go:356] "Generic (PLEG): container finished" podID="0044f242-b3e3-44d6-897b-c147c1ff6118" containerID="3ecd5f2e5ab9d2cee9304330fbd54f9e19298b27f5f890fa1ce53499220902be" exitCode=0
Oct 30 17:28:59.850191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.849759    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq" event={"ID":"0044f242-b3e3-44d6-897b-c147c1ff6118","Type":"ContainerDied","Data":"3ecd5f2e5ab9d2cee9304330fbd54f9e19298b27f5f890fa1ce53499220902be"}
Oct 30 17:28:59.863377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.863352    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"]
Oct 30 17:28:59.867131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.867113    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:28:59.897688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.896833    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"]
Oct 30 17:28:59.978618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.978587    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l89k2\" (UniqueName: \"kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:28:59.978847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.978810    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:28:59.978914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:28:59.978901    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.079737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.079702    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l89k2\" (UniqueName: \"kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.079935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.079855    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.079935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.079905    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.080658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.080620    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.082735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.082710    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.103368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.103304    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l89k2\" (UniqueName: \"kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2\") pod \"maas-api-revoke-sub-33218e82-jgxx5\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.180213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.180173    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:00.500779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.500699    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"]
Oct 30 17:29:00.505218 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:00.505189    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94229a0_5c04_40c6_9c12_bd769ec9ad2f.slice/crio-dfcf77639bf16fc0dffff5925e1ce020a828297697a8ea3ab2d41263611051d3 WatchSource:0}: Error finding container dfcf77639bf16fc0dffff5925e1ce020a828297697a8ea3ab2d41263611051d3: Status 404 returned error can't find the container with id dfcf77639bf16fc0dffff5925e1ce020a828297697a8ea3ab2d41263611051d3
Oct 30 17:29:00.662168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.662131    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"]
Oct 30 17:29:00.668367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.668345    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.680429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.680405    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"]
Oct 30 17:29:00.686942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.686916    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.687080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.686957    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.687164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.687117    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.787761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.787681    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.787761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.787746    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.787961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.787779    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.788577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.788550    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.790321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.790301    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.810628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.810602    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r\") pod \"maas-api-revoke-sub-e59c5757-qzvjg\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.858989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.858951    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5" event={"ID":"f94229a0-5c04-40c6-9c12-bd769ec9ad2f","Type":"ContainerStarted","Data":"dfcf77639bf16fc0dffff5925e1ce020a828297697a8ea3ab2d41263611051d3"}
Oct 30 17:29:00.997935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.997878    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:00.998988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:00.998966    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:29:01.083853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.080987    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"]
Oct 30 17:29:01.083853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.081794    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0044f242-b3e3-44d6-897b-c147c1ff6118" containerName="revoke-keys"
Oct 30 17:29:01.083853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.081810    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0044f242-b3e3-44d6-897b-c147c1ff6118" containerName="revoke-keys"
Oct 30 17:29:01.083853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.081989    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0044f242-b3e3-44d6-897b-c147c1ff6118" containerName="revoke-keys"
Oct 30 17:29:01.086899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.086877    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.091242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.091218    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp\" (UniqueName: \"kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp\") pod \"0044f242-b3e3-44d6-897b-c147c1ff6118\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") "
Oct 30 17:29:01.091359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.091317    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca\") pod \"0044f242-b3e3-44d6-897b-c147c1ff6118\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") "
Oct 30 17:29:01.091443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.091428    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca\") pod \"0044f242-b3e3-44d6-897b-c147c1ff6118\" (UID: \"0044f242-b3e3-44d6-897b-c147c1ff6118\") "
Oct 30 17:29:01.091764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.091747    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.091845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.091831    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvjx\" (UniqueName: \"kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.091904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.091890    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.092735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.092631    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca" pod "0044f242-b3e3-44d6-897b-c147c1ff6118" (UID: "0044f242-b3e3-44d6-897b-c147c1ff6118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:01.095799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.095549    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca" pod "0044f242-b3e3-44d6-897b-c147c1ff6118" (UID: "0044f242-b3e3-44d6-897b-c147c1ff6118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:01.095799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.095706    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp" pod "0044f242-b3e3-44d6-897b-c147c1ff6118" (UID: "0044f242-b3e3-44d6-897b-c147c1ff6118"). InnerVolumeSpecName "kube-api-access-48qbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:01.107697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.105292    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"]
Oct 30 17:29:01.131154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.131126    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:01.192695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.192657    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.192894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.192756    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvjx\" (UniqueName: \"kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.192894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.192809    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.193012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.192927    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48qbp\" (UniqueName: \"kubernetes.io/projected/0044f242-b3e3-44d6-897b-c147c1ff6118-kube-api-access-48qbp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:01.193012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.192943    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:01.193012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.192952    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0044f242-b3e3-44d6-897b-c147c1ff6118-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:01.193446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.193424    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.195279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.195257    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.217433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.217393    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvjx\" (UniqueName: \"kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx\") pod \"maas-api-revoke-sub-a6c10df0-f7vlk\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.311423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.311343    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"]
Oct 30 17:29:01.316025 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:01.315991    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb577c62_ebe9_4661_a5c4_cdcce87c9a80.slice/crio-db7d763d73b6b5a6b52ec4fa2a5664fde34972874b4576f55b17c6876330046c WatchSource:0}: Error finding container db7d763d73b6b5a6b52ec4fa2a5664fde34972874b4576f55b17c6876330046c: Status 404 returned error can't find the container with id db7d763d73b6b5a6b52ec4fa2a5664fde34972874b4576f55b17c6876330046c
Oct 30 17:29:01.406738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.406706    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:01.713084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.713039    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"]
Oct 30 17:29:01.714410 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:01.714377    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0ff7de_4dd2_449d_9fc8_5adcfc19b8ae.slice/crio-ab56b4b5e4b040edb55f41e8092ef25322f861889ee39dc3104ac8950268c270 WatchSource:0}: Error finding container ab56b4b5e4b040edb55f41e8092ef25322f861889ee39dc3104ac8950268c270: Status 404 returned error can't find the container with id ab56b4b5e4b040edb55f41e8092ef25322f861889ee39dc3104ac8950268c270
Oct 30 17:29:01.866235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.866193    5534 generic.go:356] "Generic (PLEG): container finished" podID="f94229a0-5c04-40c6-9c12-bd769ec9ad2f" containerID="c73e0e8eb33ad64657855f8bbba45849c9946d4a981a6027fc0bd9b4c2fbdd3e" exitCode=0
Oct 30 17:29:01.866727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.866288    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5" event={"ID":"f94229a0-5c04-40c6-9c12-bd769ec9ad2f","Type":"ContainerDied","Data":"c73e0e8eb33ad64657855f8bbba45849c9946d4a981a6027fc0bd9b4c2fbdd3e"}
Oct 30 17:29:01.867853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.867824    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"
Oct 30 17:29:01.867977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.867818    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq" event={"ID":"0044f242-b3e3-44d6-897b-c147c1ff6118","Type":"ContainerDied","Data":"e50fae9a905adf4ddf0fd193bbe8ceddda94ae02c87e0b87b7826370c001e390"}
Oct 30 17:29:01.867977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.867974    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50fae9a905adf4ddf0fd193bbe8ceddda94ae02c87e0b87b7826370c001e390"
Oct 30 17:29:01.869470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.869449    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg" event={"ID":"eb577c62-ebe9-4661-a5c4-cdcce87c9a80","Type":"ContainerStarted","Data":"6aa3bc994f5771de5b4ffc95b24c2a0765206ab61921e20fca49647ede5d2b25"}
Oct 30 17:29:01.869585 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.869489    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg" event={"ID":"eb577c62-ebe9-4661-a5c4-cdcce87c9a80","Type":"ContainerStarted","Data":"db7d763d73b6b5a6b52ec4fa2a5664fde34972874b4576f55b17c6876330046c"}
Oct 30 17:29:01.871160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.871136    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk" event={"ID":"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae","Type":"ContainerStarted","Data":"ab56b4b5e4b040edb55f41e8092ef25322f861889ee39dc3104ac8950268c270"}
Oct 30 17:29:01.964868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:01.964817    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg" podStartSLOduration=1.964802541 podStartE2EDuration="1.964802541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:01.963900056 +0000 UTC m=+947.725943881" watchObservedRunningTime="2026-10-30 17:29:01.964802541 +0000 UTC m=+947.726846354"
Oct 30 17:29:02.762538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:02.762500    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:02.886308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:02.886273    5534 generic.go:356] "Generic (PLEG): container finished" podID="eb577c62-ebe9-4661-a5c4-cdcce87c9a80" containerID="6aa3bc994f5771de5b4ffc95b24c2a0765206ab61921e20fca49647ede5d2b25" exitCode=0
Oct 30 17:29:02.886817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:02.886355    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg" event={"ID":"eb577c62-ebe9-4661-a5c4-cdcce87c9a80","Type":"ContainerDied","Data":"6aa3bc994f5771de5b4ffc95b24c2a0765206ab61921e20fca49647ede5d2b25"}
Oct 30 17:29:02.887965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:02.887940    5534 generic.go:356] "Generic (PLEG): container finished" podID="af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" containerID="b48e95db67e15ff24e968826839a4a892359064a69040f68651441f831858ef3" exitCode=0
Oct 30 17:29:02.888094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:02.888023    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk" event={"ID":"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae","Type":"ContainerDied","Data":"b48e95db67e15ff24e968826839a4a892359064a69040f68651441f831858ef3"}
Oct 30 17:29:03.055393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.055361    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:03.127478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.127442    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca\") pod \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") "
Oct 30 17:29:03.127691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.127493    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2\" (UniqueName: \"kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2\") pod \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") "
Oct 30 17:29:03.127691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.127550    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca\") pod \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\" (UID: \"f94229a0-5c04-40c6-9c12-bd769ec9ad2f\") "
Oct 30 17:29:03.127936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.127908    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca" pod "f94229a0-5c04-40c6-9c12-bd769ec9ad2f" (UID: "f94229a0-5c04-40c6-9c12-bd769ec9ad2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:03.128072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.128035    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:03.129853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.129824    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca" pod "f94229a0-5c04-40c6-9c12-bd769ec9ad2f" (UID: "f94229a0-5c04-40c6-9c12-bd769ec9ad2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:03.130127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.130095    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2" pod "f94229a0-5c04-40c6-9c12-bd769ec9ad2f" (UID: "f94229a0-5c04-40c6-9c12-bd769ec9ad2f"). InnerVolumeSpecName "kube-api-access-l89k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:03.229126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.229032    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:03.229126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.229067    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l89k2\" (UniqueName: \"kubernetes.io/projected/f94229a0-5c04-40c6-9c12-bd769ec9ad2f-kube-api-access-l89k2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:03.896676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.896630    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"
Oct 30 17:29:03.897871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.896668    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5" event={"ID":"f94229a0-5c04-40c6-9c12-bd769ec9ad2f","Type":"ContainerDied","Data":"dfcf77639bf16fc0dffff5925e1ce020a828297697a8ea3ab2d41263611051d3"}
Oct 30 17:29:03.898001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:03.897905    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfcf77639bf16fc0dffff5925e1ce020a828297697a8ea3ab2d41263611051d3"
Oct 30 17:29:04.093726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.093697    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:04.097027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.097001    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:04.137136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137105    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx\" (UniqueName: \"kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx\") pod \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") "
Oct 30 17:29:04.137312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137206    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca\") pod \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") "
Oct 30 17:29:04.137312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137239    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r\") pod \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") "
Oct 30 17:29:04.137312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137296    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca\") pod \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") "
Oct 30 17:29:04.137738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137354    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca\") pod \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\" (UID: \"eb577c62-ebe9-4661-a5c4-cdcce87c9a80\") "
Oct 30 17:29:04.137738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137460    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca\") pod \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\" (UID: \"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae\") "
Oct 30 17:29:04.137860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137760    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca" pod "eb577c62-ebe9-4661-a5c4-cdcce87c9a80" (UID: "eb577c62-ebe9-4661-a5c4-cdcce87c9a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:04.137949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137921    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca" pod "af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" (UID: "af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:04.137949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.137936    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.140461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.140431    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx" pod "af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" (UID: "af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae"). InnerVolumeSpecName "kube-api-access-ssvjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:04.140789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.140761    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca" pod "af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" (UID: "af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:04.140957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.140936    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r" pod "eb577c62-ebe9-4661-a5c4-cdcce87c9a80" (UID: "eb577c62-ebe9-4661-a5c4-cdcce87c9a80"). InnerVolumeSpecName "kube-api-access-2gp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:04.141018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.141008    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca" pod "eb577c62-ebe9-4661-a5c4-cdcce87c9a80" (UID: "eb577c62-ebe9-4661-a5c4-cdcce87c9a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:04.238565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.238531    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.238565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.238564    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.238793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.238574    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvjx\" (UniqueName: \"kubernetes.io/projected/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-kube-api-access-ssvjx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.238793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.238583    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.238793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.238593    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/eb577c62-ebe9-4661-a5c4-cdcce87c9a80-kube-api-access-2gp6r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.510658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.510548    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:04.909200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.909163    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg" event={"ID":"eb577c62-ebe9-4661-a5c4-cdcce87c9a80","Type":"ContainerDied","Data":"db7d763d73b6b5a6b52ec4fa2a5664fde34972874b4576f55b17c6876330046c"}
Oct 30 17:29:04.909200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.909174    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"
Oct 30 17:29:04.909200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.909206    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7d763d73b6b5a6b52ec4fa2a5664fde34972874b4576f55b17c6876330046c"
Oct 30 17:29:04.910614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.910591    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"
Oct 30 17:29:04.910762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.910610    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk" event={"ID":"af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae","Type":"ContainerDied","Data":"ab56b4b5e4b040edb55f41e8092ef25322f861889ee39dc3104ac8950268c270"}
Oct 30 17:29:04.910762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:04.910690    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab56b4b5e4b040edb55f41e8092ef25322f861889ee39dc3104ac8950268c270"
Oct 30 17:29:06.314145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:06.314107    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:07.796854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.796818    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"]
Oct 30 17:29:07.797320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797306    5534 cpu_manager.go:414] "containerMap: removing container" podUID="eb577c62-ebe9-4661-a5c4-cdcce87c9a80" containerName="revoke-keys"
Oct 30 17:29:07.797320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797320    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb577c62-ebe9-4661-a5c4-cdcce87c9a80" containerName="revoke-keys"
Oct 30 17:29:07.797399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797330    5534 cpu_manager.go:414] "containerMap: removing container" podUID="af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" containerName="revoke-keys"
Oct 30 17:29:07.797399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797335    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" containerName="revoke-keys"
Oct 30 17:29:07.797399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797365    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f94229a0-5c04-40c6-9c12-bd769ec9ad2f" containerName="revoke-keys"
Oct 30 17:29:07.797399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797372    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94229a0-5c04-40c6-9c12-bd769ec9ad2f" containerName="revoke-keys"
Oct 30 17:29:07.797535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797454    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb577c62-ebe9-4661-a5c4-cdcce87c9a80" containerName="revoke-keys"
Oct 30 17:29:07.797535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797462    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" containerName="revoke-keys"
Oct 30 17:29:07.797535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.797473    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94229a0-5c04-40c6-9c12-bd769ec9ad2f" containerName="revoke-keys"
Oct 30 17:29:07.800996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.800978    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.824841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.824810    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"]
Oct 30 17:29:07.832616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.832587    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0e7403-openshift-default-dockercfg-z9pr5\""
Oct 30 17:29:07.882407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.882364    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.882596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.882450    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mm5s\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.882738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.882597    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.882807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.882790    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.882867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.882835    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.882950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.882930    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.883083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.883065    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.883147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.883099    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.883205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.883177    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.984503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984467    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.984503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984505    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.984775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984581    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.984775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984601    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.984775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984626    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.984936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984678    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.984988    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mm5s\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985035    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985045    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985069    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985105    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985125    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985363    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.985824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.985799    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.987489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.987460    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:07.987657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:07.987618    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:08.007775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.007747    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:08.007919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.007897    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mm5s\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:08.114947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.114914    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:08.205516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.205484    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:08.437127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.437105    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"]
Oct 30 17:29:08.438553 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:08.438523    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c435b47_b991_46a7_a936_97c7b25fd6ba.slice/crio-8de45dc2be3af66328c30ab7e2c872b922ded93aae9b3966eae28b387d22e4a4 WatchSource:0}: Error finding container 8de45dc2be3af66328c30ab7e2c872b922ded93aae9b3966eae28b387d22e4a4: Status 404 returned error can't find the container with id 8de45dc2be3af66328c30ab7e2c872b922ded93aae9b3966eae28b387d22e4a4
Oct 30 17:29:08.440838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.440808    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:08.440898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.440879    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:08.440932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.440912    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:08.935268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.935224    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" event={"ID":"3c435b47-b991-46a7-a936-97c7b25fd6ba","Type":"ContainerStarted","Data":"8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d"}
Oct 30 17:29:08.935268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.935276    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" event={"ID":"3c435b47-b991-46a7-a936-97c7b25fd6ba","Type":"ContainerStarted","Data":"8de45dc2be3af66328c30ab7e2c872b922ded93aae9b3966eae28b387d22e4a4"}
Oct 30 17:29:08.982756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:08.982691    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podStartSLOduration=1.982670529 podStartE2EDuration="1.982670529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:08.980180267 +0000 UTC m=+954.742224082" watchObservedRunningTime="2026-10-30 17:29:08.982670529 +0000 UTC m=+954.744714341"
Oct 30 17:29:09.115546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:09.115517    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:09.117496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:09.117469    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:29:09.117665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:09.117516    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:29:10.084057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:10.084005    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:10.115763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:10.115735    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:29:10.115912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:10.115788    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:29:11.115756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:11.115721    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:29:11.116179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:11.115785    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:29:11.706439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:11.706404    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:12.116239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.116206    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:29:12.116675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.116267    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:29:12.632021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.631977    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"]
Oct 30 17:29:12.636525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.636497    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.682237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.682200    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"]
Oct 30 17:29:12.747176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747129    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747262    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747299    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747391    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747491    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747519    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747565    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747704    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.747789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.747753    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhn6p\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.848660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848617    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.848829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848695    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.848829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848716    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.848829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848736    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.848829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848795    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.848829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848821    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhn6p\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848846    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848886    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.848909    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.849079    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.849378    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.849391    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.849441    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.849881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.849859    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.851243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.851224    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.851479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.851462    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.885903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.885797    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.886351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.886321    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhn6p\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p\") pod \"maas-default-gateway-openshift-default-7df854bf67-g5nm8\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:12.956918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:12.956879    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:13.116991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.116953    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:29:13.117446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.117003    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:29:13.308108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.308081    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"]
Oct 30 17:29:13.308596 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:13.308571    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803484b3_0ab3_403f_9f7f_152d7da42061.slice/crio-a2619a6c72a05c36627aa08f067ed529c4bd6dffc49a4f7d8e82d61a1ba6c033 WatchSource:0}: Error finding container a2619a6c72a05c36627aa08f067ed529c4bd6dffc49a4f7d8e82d61a1ba6c033: Status 404 returned error can't find the container with id a2619a6c72a05c36627aa08f067ed529c4bd6dffc49a4f7d8e82d61a1ba6c033
Oct 30 17:29:13.311037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.311004    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:13.311136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.311071    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:13.311136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.311105    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:13.782310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.782272    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:13.892955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.892922    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"]
Oct 30 17:29:13.896997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.896976    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:13.911857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.911833    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:29:13.912746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.912720    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"]
Oct 30 17:29:13.920147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:13.920121    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:29:14.005298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.005267    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jhd\" (UniqueName: \"kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.005662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.005616    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.005936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.005917    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.021513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.014298    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" event={"ID":"803484b3-0ab3-403f-9f7f-152d7da42061","Type":"ContainerStarted","Data":"4e42df9cded89e52f8a50c3c1d781e06bc5fc0d18fd7072334580cae637b71a6"}
Oct 30 17:29:14.021876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.021847    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" event={"ID":"803484b3-0ab3-403f-9f7f-152d7da42061","Type":"ContainerStarted","Data":"a2619a6c72a05c36627aa08f067ed529c4bd6dffc49a4f7d8e82d61a1ba6c033"}
Oct 30 17:29:14.106716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.094226    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podStartSLOduration=2.094203668 podStartE2EDuration="2.094203668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:14.093007619 +0000 UTC m=+959.855051433" watchObservedRunningTime="2026-10-30 17:29:14.094203668 +0000 UTC m=+959.856247482"
Oct 30 17:29:14.109666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.109523    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.110536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.109865    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jhd\" (UniqueName: \"kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.110536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.110096    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.110536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.110442    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.113696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.113668    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.124459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.124438    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:29:14.124821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.124706    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:29:14.134397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.134375    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jhd\" (UniqueName: \"kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd\") pod \"maas-api-revoke-sub-0a0e8bf1-74zgv\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.226202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.226163    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:14.594101 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:14.594054    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e8ed3e_0e89_4fb3_83cf_be44cd5374d9.slice/crio-78907d0e8f994331c8e6ea46db7b1dbc80ecb742a400d5b2aa162f1ce443d589 WatchSource:0}: Error finding container 78907d0e8f994331c8e6ea46db7b1dbc80ecb742a400d5b2aa162f1ce443d589: Status 404 returned error can't find the container with id 78907d0e8f994331c8e6ea46db7b1dbc80ecb742a400d5b2aa162f1ce443d589
Oct 30 17:29:14.595026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.594999    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"]
Oct 30 17:29:14.961982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.961952    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:14.965768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.965740    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:14.965996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:14.965967    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:15.027084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.027048    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv" event={"ID":"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9","Type":"ContainerStarted","Data":"ad0935354fe95e76bc867cd6f3124dd2862ccd21107bbdd223305452ddd90c64"}
Oct 30 17:29:15.027232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.027112    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv" event={"ID":"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9","Type":"ContainerStarted","Data":"78907d0e8f994331c8e6ea46db7b1dbc80ecb742a400d5b2aa162f1ce443d589"}
Oct 30 17:29:15.113671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.104910    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"]
Oct 30 17:29:15.114164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.114110    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv" podStartSLOduration=2.114090143 podStartE2EDuration="2.114090143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:15.080050992 +0000 UTC m=+960.842094806" watchObservedRunningTime="2026-10-30 17:29:15.114090143 +0000 UTC m=+960.876133958"
Oct 30 17:29:15.121035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.121010    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"]
Oct 30 17:29:15.121307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.121264    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.135592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.135506    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:15.137893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.136732    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"distinct-w1-455086-kserve-self-signed-certs\""
Oct 30 17:29:15.238899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.238701    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.239063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.238895    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zzf\" (UniqueName: \"kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.239063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.238930    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.239063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.238976    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.239246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.239104    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.239246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.239195    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.248419 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:29:15.248391    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:29:15.341695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.341190    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44zzf\" (UniqueName: \"kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.341695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.341276    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.341695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.341358    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.341695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.341427    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.341695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.341513    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.341695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.341635    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.342511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.342478    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.342946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.342923    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.343060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.342938    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.346282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.346266    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.349045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.349015    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.372372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.372341    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zzf\" (UniqueName: \"kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf\") pod \"distinct-w1-455086-kserve-786fcff9d8-nd2g8\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") " pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.498713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.498275    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:15.532668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.532610    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:15.534302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.534191    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:15.535368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.535318    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:29:15.779634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.779542    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:15.839130 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:15.839098    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda53ac57c_cc71_4a25_8d14_97b27b9dedd6.slice/crio-03ab51f877174d21095bda7b3d7fb560a0a51ff669e96ccfe76f3dfb8dad4496 WatchSource:0}: Error finding container 03ab51f877174d21095bda7b3d7fb560a0a51ff669e96ccfe76f3dfb8dad4496: Status 404 returned error can't find the container with id 03ab51f877174d21095bda7b3d7fb560a0a51ff669e96ccfe76f3dfb8dad4496
Oct 30 17:29:15.839267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.839187    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"]
Oct 30 17:29:15.975185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.975156    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:15.975423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:15.975392    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:16.034831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.034749    5534 generic.go:356] "Generic (PLEG): container finished" podID="d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" containerID="ad0935354fe95e76bc867cd6f3124dd2862ccd21107bbdd223305452ddd90c64" exitCode=0
Oct 30 17:29:16.034831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.034803    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv" event={"ID":"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9","Type":"ContainerDied","Data":"ad0935354fe95e76bc867cd6f3124dd2862ccd21107bbdd223305452ddd90c64"}
Oct 30 17:29:16.036534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.036511    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" event={"ID":"a53ac57c-cc71-4a25-8d14-97b27b9dedd6","Type":"ContainerStarted","Data":"ed4209b545eb1c0749623bf68087118239db59ae85df7a4ff51138da71f2a785"}
Oct 30 17:29:16.036626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.036546    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" event={"ID":"a53ac57c-cc71-4a25-8d14-97b27b9dedd6","Type":"ContainerStarted","Data":"03ab51f877174d21095bda7b3d7fb560a0a51ff669e96ccfe76f3dfb8dad4496"}
Oct 30 17:29:16.122210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.122151    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podStartSLOduration=1.122132696 podStartE2EDuration="1.122132696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:16.119168661 +0000 UTC m=+961.881212475" watchObservedRunningTime="2026-10-30 17:29:16.122132696 +0000 UTC m=+961.884176510"
Oct 30 17:29:16.498964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.498935    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:16.978669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.971974    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:16.978669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:16.972035    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:17.249947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.249923    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:17.397869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.397829    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca\") pod \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") "
Oct 30 17:29:17.398073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.397882    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd\" (UniqueName: \"kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd\") pod \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") "
Oct 30 17:29:17.398073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.397924    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca\") pod \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\" (UID: \"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9\") "
Oct 30 17:29:17.398884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.398853    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca" pod "d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" (UID: "d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:17.400971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.400938    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca" pod "d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" (UID: "d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:17.401809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.401782    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd" pod "d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" (UID: "d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9"). InnerVolumeSpecName "kube-api-access-w5jhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:17.499772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.499680    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:17.499772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.499720    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5jhd\" (UniqueName: \"kubernetes.io/projected/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-kube-api-access-w5jhd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:17.499772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.499735    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:17.678669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.664122    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:17.965414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.964982    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:17.965414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:17.965043    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:18.048949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:18.048915    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"
Oct 30 17:29:18.048949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:18.048932    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv" event={"ID":"d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9","Type":"ContainerDied","Data":"78907d0e8f994331c8e6ea46db7b1dbc80ecb742a400d5b2aa162f1ce443d589"}
Oct 30 17:29:18.049171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:18.048984    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78907d0e8f994331c8e6ea46db7b1dbc80ecb742a400d5b2aa162f1ce443d589"
Oct 30 17:29:18.965074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:18.963031    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:18.965074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:18.963107    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:19.467085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:19.467021    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:19.959929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:19.959895    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:19.960137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:19.959958    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:20.959992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:20.959957    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:29:20.960523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:20.960027    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:29:21.097699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:21.097662    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:22.014730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:22.014696    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:22.015203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:22.015083    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:22.016032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:22.016010    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 17:29:22.964109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:22.964076    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:23.181409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.181361    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"]
Oct 30 17:29:23.197674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.197618    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" containerName="revoke-keys"
Oct 30 17:29:23.197915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.197896    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" containerName="revoke-keys"
Oct 30 17:29:23.198187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.198170    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" containerName="revoke-keys"
Oct 30 17:29:23.209004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.208973    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.227732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.225764    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"]
Oct 30 17:29:23.230547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.229749    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-506124-openshift-default-dockercfg-xdvhh\""
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391377    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391441    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391505    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391557    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391615    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391668    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391699    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391730    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qjj\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.393670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.391760    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498660    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498746    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498784    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498817    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498848    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qjj\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498875    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.498998    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.499037    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.499713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.499101    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.500302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.499960    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.517810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.500695    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.517810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.500948    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.517810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.501180    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.517810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.501420    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.517810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.503481    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.541709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.538019    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.548682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.548595    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.550761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.550734    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qjj\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:23.839603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:23.839506    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:24.171002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.170972    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"]
Oct 30 17:29:24.173713 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:24.173682    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c62976b_8f4a_4b0d_929c_7dcaf375911c.slice/crio-c1b494d83080383d259bca0731b0915f18c95b268d54cce27eaf4a0c1aea8c4f WatchSource:0}: Error finding container c1b494d83080383d259bca0731b0915f18c95b268d54cce27eaf4a0c1aea8c4f: Status 404 returned error can't find the container with id c1b494d83080383d259bca0731b0915f18c95b268d54cce27eaf4a0c1aea8c4f
Oct 30 17:29:24.176527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.176493    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:24.176599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.176577    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:24.176659 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.176617    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:24.320490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.320457    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"]
Oct 30 17:29:24.325698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.325674    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.340915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.340661    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:29:24.340915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.340738    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:29:24.344896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.344872    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"]
Oct 30 17:29:24.425723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.425072    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ztt\" (UniqueName: \"kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.425723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.425282    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.425723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.425335    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.526685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.526597    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.526907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.526733    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.526907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.526872    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ztt\" (UniqueName: \"kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.527371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.527345    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.529561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.529535    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.550826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.550800    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ztt\" (UniqueName: \"kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt\") pod \"maas-api-revoke-sub-8eadb3ac-xxd78\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.645366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.645333    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:24.826864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.826820    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:24.961442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:24.961418    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"]
Oct 30 17:29:24.964788 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:24.964697    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a8d3d5_15d8_4e09_b9af_f61c3f571336.slice/crio-e3fe5fbffb7acb68ac472e0d89aac0f4acec92104fcc9380d8ebf7e516fd7914 WatchSource:0}: Error finding container e3fe5fbffb7acb68ac472e0d89aac0f4acec92104fcc9380d8ebf7e516fd7914: Status 404 returned error can't find the container with id e3fe5fbffb7acb68ac472e0d89aac0f4acec92104fcc9380d8ebf7e516fd7914
Oct 30 17:29:25.113602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.113564    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78" event={"ID":"90a8d3d5-15d8-4e09-b9af-f61c3f571336","Type":"ContainerStarted","Data":"e3fe5fbffb7acb68ac472e0d89aac0f4acec92104fcc9380d8ebf7e516fd7914"}
Oct 30 17:29:25.115380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.115353    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" event={"ID":"1c62976b-8f4a-4b0d-929c-7dcaf375911c","Type":"ContainerStarted","Data":"2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d"}
Oct 30 17:29:25.115519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.115399    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" event={"ID":"1c62976b-8f4a-4b0d-929c-7dcaf375911c","Type":"ContainerStarted","Data":"c1b494d83080383d259bca0731b0915f18c95b268d54cce27eaf4a0c1aea8c4f"}
Oct 30 17:29:25.164669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.163152    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podStartSLOduration=2.163130439 podStartE2EDuration="2.163130439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:25.163044736 +0000 UTC m=+970.925088550" watchObservedRunningTime="2026-10-30 17:29:25.163130439 +0000 UTC m=+970.925174256"
Oct 30 17:29:25.501666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.499431    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:25.514844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.514814    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:25.840147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.840122    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:25.841696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.841672    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:29:25.841787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:25.841725    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:29:26.124948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:26.123993    5534 generic.go:356] "Generic (PLEG): container finished" podID="90a8d3d5-15d8-4e09-b9af-f61c3f571336" containerID="4c4f4e419ef484e9ea24c285796d75ad26a35cf7997c752e97d9e23a8bd1b89f" exitCode=0
Oct 30 17:29:26.124948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:26.124072    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78" event={"ID":"90a8d3d5-15d8-4e09-b9af-f61c3f571336","Type":"ContainerDied","Data":"4c4f4e419ef484e9ea24c285796d75ad26a35cf7997c752e97d9e23a8bd1b89f"}
Oct 30 17:29:26.136347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:26.136325    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:29:26.606745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:26.606716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:26.840735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:26.840699    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:29:26.840932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:26.840762    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:29:27.327742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.327707    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:27.392492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.392406    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca\") pod \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") "
Oct 30 17:29:27.392492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.392479    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt\" (UniqueName: \"kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt\") pod \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") "
Oct 30 17:29:27.393345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.393314    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca\") pod \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\" (UID: \"90a8d3d5-15d8-4e09-b9af-f61c3f571336\") "
Oct 30 17:29:27.393859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.393834    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca" pod "90a8d3d5-15d8-4e09-b9af-f61c3f571336" (UID: "90a8d3d5-15d8-4e09-b9af-f61c3f571336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:27.394161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.394142    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:27.395976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.395667    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca" pod "90a8d3d5-15d8-4e09-b9af-f61c3f571336" (UID: "90a8d3d5-15d8-4e09-b9af-f61c3f571336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:27.398607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.398580    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt" pod "90a8d3d5-15d8-4e09-b9af-f61c3f571336" (UID: "90a8d3d5-15d8-4e09-b9af-f61c3f571336"). InnerVolumeSpecName "kube-api-access-b2ztt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:27.495495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.495451    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a8d3d5-15d8-4e09-b9af-f61c3f571336-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:27.495495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.495483    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ztt\" (UniqueName: \"kubernetes.io/projected/90a8d3d5-15d8-4e09-b9af-f61c3f571336-kube-api-access-b2ztt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:27.840275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.840247    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:29:27.840673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:27.840304    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:29:28.138696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.138601    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"
Oct 30 17:29:28.138856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.138618    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78" event={"ID":"90a8d3d5-15d8-4e09-b9af-f61c3f571336","Type":"ContainerDied","Data":"e3fe5fbffb7acb68ac472e0d89aac0f4acec92104fcc9380d8ebf7e516fd7914"}
Oct 30 17:29:28.138974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.138962    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3fe5fbffb7acb68ac472e0d89aac0f4acec92104fcc9380d8ebf7e516fd7914"
Oct 30 17:29:28.450246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.448367    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:28.834591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.834496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:28.840663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.840612    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:29:28.841059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.840687    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:29:28.931391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.931359    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"]
Oct 30 17:29:28.931993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.931978    5534 cpu_manager.go:414] "containerMap: removing container" podUID="90a8d3d5-15d8-4e09-b9af-f61c3f571336" containerName="revoke-keys"
Oct 30 17:29:28.932054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.931995    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a8d3d5-15d8-4e09-b9af-f61c3f571336" containerName="revoke-keys"
Oct 30 17:29:28.932093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.932083    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a8d3d5-15d8-4e09-b9af-f61c3f571336" containerName="revoke-keys"
Oct 30 17:29:28.935194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.935175    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:28.978667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.977668    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"]
Oct 30 17:29:28.978667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.977812    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:29:28.983910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:28.983885    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:29:29.018276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.018246    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt69r\" (UniqueName: \"kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.018407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.018304    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.018538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.018516    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.120110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.120081    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.120317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.120152    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt69r\" (UniqueName: \"kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.120317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.120182    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.120899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.120835    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.122582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.122563    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.178723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.178342    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt69r\" (UniqueName: \"kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r\") pod \"maas-api-revoke-sub-9bd1f492-w58vs\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.248044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.248005    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:29.586587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.586558    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"]
Oct 30 17:29:29.587565 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:29.587531    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dfa281_33d8_476a_a736_fa65f964602a.slice/crio-5b59367de2be01b9185b44586b1403fc6966635d9ebde28795bd53c76734f6fe WatchSource:0}: Error finding container 5b59367de2be01b9185b44586b1403fc6966635d9ebde28795bd53c76734f6fe: Status 404 returned error can't find the container with id 5b59367de2be01b9185b44586b1403fc6966635d9ebde28795bd53c76734f6fe
Oct 30 17:29:29.840356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.840331    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:29:29.840504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:29.840384    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:29:30.152247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:30.152163    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs" event={"ID":"f9dfa281-33d8-476a-a736-fa65f964602a","Type":"ContainerStarted","Data":"587e7b0cb22b968268ec6a199c9829c4e44876f163999fc531ef0835082e6adb"}
Oct 30 17:29:30.152247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:30.152218    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs" event={"ID":"f9dfa281-33d8-476a-a736-fa65f964602a","Type":"ContainerStarted","Data":"5b59367de2be01b9185b44586b1403fc6966635d9ebde28795bd53c76734f6fe"}
Oct 30 17:29:30.193987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:30.193936    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs" podStartSLOduration=2.193923312 podStartE2EDuration="2.193923312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:30.192361367 +0000 UTC m=+975.954405238" watchObservedRunningTime="2026-10-30 17:29:30.193923312 +0000 UTC m=+975.955967134"
Oct 30 17:29:30.342127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:30.342092    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:30.840788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:30.840752    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:29:30.840960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:30.840815    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:29:31.160967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:31.160878    5534 generic.go:356] "Generic (PLEG): container finished" podID="f9dfa281-33d8-476a-a736-fa65f964602a" containerID="587e7b0cb22b968268ec6a199c9829c4e44876f163999fc531ef0835082e6adb" exitCode=0
Oct 30 17:29:31.160967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:31.160926    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs" event={"ID":"f9dfa281-33d8-476a-a736-fa65f964602a","Type":"ContainerDied","Data":"587e7b0cb22b968268ec6a199c9829c4e44876f163999fc531ef0835082e6adb"}
Oct 30 17:29:31.844895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:31.844865    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:31.845383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:31.845370    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:31.846097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:31.846077    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:29:32.230734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.226664    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:32.340494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.340470    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:32.465411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.465326    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r\" (UniqueName: \"kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r\") pod \"f9dfa281-33d8-476a-a736-fa65f964602a\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") "
Oct 30 17:29:32.465411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.465402    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca\") pod \"f9dfa281-33d8-476a-a736-fa65f964602a\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") "
Oct 30 17:29:32.465632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.465524    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca\") pod \"f9dfa281-33d8-476a-a736-fa65f964602a\" (UID: \"f9dfa281-33d8-476a-a736-fa65f964602a\") "
Oct 30 17:29:32.466126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.465922    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca" pod "f9dfa281-33d8-476a-a736-fa65f964602a" (UID: "f9dfa281-33d8-476a-a736-fa65f964602a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:32.466126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.466088    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:32.467725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.467699    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r" pod "f9dfa281-33d8-476a-a736-fa65f964602a" (UID: "f9dfa281-33d8-476a-a736-fa65f964602a"). InnerVolumeSpecName "kube-api-access-mt69r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:32.467839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.467699    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca" pod "f9dfa281-33d8-476a-a736-fa65f964602a" (UID: "f9dfa281-33d8-476a-a736-fa65f964602a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:32.567602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.567561    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt69r\" (UniqueName: \"kubernetes.io/projected/f9dfa281-33d8-476a-a736-fa65f964602a-kube-api-access-mt69r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:32.567602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:32.567597    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfa281-33d8-476a-a736-fa65f964602a-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:29:33.178900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:33.178867    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"
Oct 30 17:29:33.178900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:33.178890    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs" event={"ID":"f9dfa281-33d8-476a-a736-fa65f964602a","Type":"ContainerDied","Data":"5b59367de2be01b9185b44586b1403fc6966635d9ebde28795bd53c76734f6fe"}
Oct 30 17:29:33.179154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:33.178951    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b59367de2be01b9185b44586b1403fc6966635d9ebde28795bd53c76734f6fe"
Oct 30 17:29:33.944364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:33.944334    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:35.595347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:35.595301    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:37.516472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.516433    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:37.661937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.661897    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:37.824480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.824392    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"]
Oct 30 17:29:37.825081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.825061    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f9dfa281-33d8-476a-a736-fa65f964602a" containerName="revoke-keys"
Oct 30 17:29:37.825183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.825082    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9dfa281-33d8-476a-a736-fa65f964602a" containerName="revoke-keys"
Oct 30 17:29:37.825272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.825259    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9dfa281-33d8-476a-a736-fa65f964602a" containerName="revoke-keys"
Oct 30 17:29:37.829898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.829878    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.852777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.852740    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"]
Oct 30 17:29:37.933412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933375    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933516    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933556    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933612    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933815    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933867    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933898    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:37.933936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:37.933927    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nkcb\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.034968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.034932    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.034968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.034967    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035009    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035034    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035083    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035418    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035486    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035501    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035521    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nkcb\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035543    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035670    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.035895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.035874    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.036049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.036030    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.036329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.036310    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.037966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.037941    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.038214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.038197    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.057897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.057874    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.058081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.058064    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nkcb\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.145664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.145604    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:38.469989 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:38.469956    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9ce9aa_bb0f_494e_805a_eaa70fc5b873.slice/crio-61e7b1d4b88985e58e81890b1ad39fd8acb06268ca299ef0ff20f6ac0f1841f2 WatchSource:0}: Error finding container 61e7b1d4b88985e58e81890b1ad39fd8acb06268ca299ef0ff20f6ac0f1841f2: Status 404 returned error can't find the container with id 61e7b1d4b88985e58e81890b1ad39fd8acb06268ca299ef0ff20f6ac0f1841f2
Oct 30 17:29:38.470842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.470811    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"]
Oct 30 17:29:38.473002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.472970    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:38.473091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.473034    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:38.473091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.473062    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:38.998195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:38.998152    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:39.231912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:39.231868    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" event={"ID":"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873","Type":"ContainerStarted","Data":"c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a"}
Oct 30 17:29:39.232061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:39.231966    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" event={"ID":"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873","Type":"ContainerStarted","Data":"61e7b1d4b88985e58e81890b1ad39fd8acb06268ca299ef0ff20f6ac0f1841f2"}
Oct 30 17:29:39.279705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:39.279592    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:39.301069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:39.301015    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podStartSLOduration=2.3009997589999998 podStartE2EDuration="2.300999759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:39.299106814 +0000 UTC m=+985.061150626" watchObservedRunningTime="2026-10-30 17:29:39.300999759 +0000 UTC m=+985.063043572"
Oct 30 17:29:40.146663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:40.146608    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:40.148051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:40.148027    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:29:40.148130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:40.148087    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:29:41.146670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:41.146607    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:29:41.147109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:41.146698    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:29:41.230966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:41.230927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:42.146212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:42.146176    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:29:42.146368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:42.146247    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:29:43.029895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:43.029857    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:43.146164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:43.146127    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:29:43.146333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:43.146205    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:29:43.889006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:43.888975    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:44.000784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:44.000743    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:44.146792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:44.146706    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:29:44.146792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:44.146766    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:29:44.843876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:44.843845    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:45.145928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:45.145839    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:29:45.145928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:45.145891    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:29:45.244177 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:29:45.244130    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:29:46.151006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:46.150974    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:46.151526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:46.151499    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:46.152288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:46.152269    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:29:46.627227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:46.627193    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:48.305951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:48.305914    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:50.120462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:50.120421    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:51.850070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:51.850036    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:53.089885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.089845    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"]
Oct 30 17:29:53.096518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.096495    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.123474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.123447    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"]
Oct 30 17:29:53.163671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.163628    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.163823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.163713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.163823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.163805    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.163920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.163847    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.164020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.163992    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.164079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.164048    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8tg\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.164176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.164158    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.164225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.164201    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.164316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.164284    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265180    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265261    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265306    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265497    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265592    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265636    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265633    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8tg\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265692    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265791    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265830    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265868    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.265932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.265882    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.266171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.266151    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.266832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.266809    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.268025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.268008    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.268130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.268113    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.289333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.289309    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.289804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.289787    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8tg\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.414578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.414484    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:53.450933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.450905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:53.734594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.734567    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"]
Oct 30 17:29:53.735770 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:29:53.735745    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6946f94e_959d_46c4_a9c9_52b17b1cb7c9.slice/crio-eb6a24af527ceb58926fc10e433912dab8a1274c4847e53ba920fee9123ed344 WatchSource:0}: Error finding container eb6a24af527ceb58926fc10e433912dab8a1274c4847e53ba920fee9123ed344: Status 404 returned error can't find the container with id eb6a24af527ceb58926fc10e433912dab8a1274c4847e53ba920fee9123ed344
Oct 30 17:29:53.738361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.738331    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:53.738475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.738410    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:53.738475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:53.738458    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:29:54.389904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:54.389823    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" event={"ID":"6946f94e-959d-46c4-a9c9-52b17b1cb7c9","Type":"ContainerStarted","Data":"b273887164b2133519e5d4837016458a8cbd8ac43d7a0866d5b081fe4fa44759"}
Oct 30 17:29:54.389904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:54.389875    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" event={"ID":"6946f94e-959d-46c4-a9c9-52b17b1cb7c9","Type":"ContainerStarted","Data":"eb6a24af527ceb58926fc10e433912dab8a1274c4847e53ba920fee9123ed344"}
Oct 30 17:29:54.414898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:54.414869    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:29:54.416633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:54.416602    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:29:54.416783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:54.416685    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:29:54.433228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:54.433179    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podStartSLOduration=1.433165656 podStartE2EDuration="1.433165656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:54.430372596 +0000 UTC m=+1000.192416409" watchObservedRunningTime="2026-10-30 17:29:54.433165656 +0000 UTC m=+1000.195209526"
Oct 30 17:29:55.415557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:55.415506    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:29:55.416076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:55.415570    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:29:55.433672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:55.433629    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:56.415203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:56.415160    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:29:56.415407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:56.415223    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:29:57.231835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:57.231786    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:57.415211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:57.415175    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:29:57.415412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:57.415240    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:29:58.415217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:58.415180    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:29:58.415630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:58.415231    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:29:58.950439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:58.950399    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:29:59.414817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:59.414787    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:29:59.414994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:29:59.414831    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:30:00.176668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.176587    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"]
Oct 30 17:30:00.181544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.181522    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"]
Oct 30 17:30:00.181704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.181681    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.186060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.186036    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"]
Oct 30 17:30:00.186195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.186169    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.191289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.190623    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.208410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.204637    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"]
Oct 30 17:30:00.211917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.209709    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:30:00.211917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.210223    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"]
Oct 30 17:30:00.216709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.216628    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.223011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.222815    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"]
Oct 30 17:30:00.223011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.222862    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"]
Oct 30 17:30:00.223011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.222893    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"]
Oct 30 17:30:00.223011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.222982    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.250181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.250154    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"]
Oct 30 17:30:00.253695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.253633    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"]
Oct 30 17:30:00.275773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275732    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5f2b\" (UniqueName: \"kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.275936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275817    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.275936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275836    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbm6\" (UniqueName: \"kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.275936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275868    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.275936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275907    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwrm\" (UniqueName: \"kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm\") pod \"maas-api-key-cleanup-29889690-fvjhp\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.276157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275945    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-fvjhp\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.276157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.275986    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.276157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.276017    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgk8\" (UniqueName: \"kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.276157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.276038    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hffb\" (UniqueName: \"kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.276157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.276062    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.377668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377403    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.377668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377471    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgk8\" (UniqueName: \"kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.377668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377511    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hffb\" (UniqueName: \"kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.377668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377549    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.377668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377622    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f2b\" (UniqueName: \"kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.378062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377738    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.378062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377767    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbm6\" (UniqueName: \"kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.378062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377815    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.378062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377879    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwrm\" (UniqueName: \"kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm\") pod \"maas-api-key-cleanup-29889690-fvjhp\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.378062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.377947    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-fvjhp\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.379702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.378789    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-fvjhp\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.379702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.379421    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.380568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.380543    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.381545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.381516    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.381775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.381753    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.405162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.405127    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgk8\" (UniqueName: \"kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.408795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.407912    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hffb\" (UniqueName: \"kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.410747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.409417    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f2b\" (UniqueName: \"kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.410747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.409867    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwrm\" (UniqueName: \"kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm\") pod \"maas-api-key-cleanup-29889690-fvjhp\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.410747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.410241    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbm6\" (UniqueName: \"kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.499048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.498966    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:00.525051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.525021    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:00.534345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.534325    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:00.549242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.549049    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:00.589320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.589089    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:00.707756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.707721    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:30:00.722191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.722050    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:30:00.725366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.725305    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:30:00.876543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:00.876501    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:01.057573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.057548    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"]
Oct 30 17:30:01.077211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.077184    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"]
Oct 30 17:30:01.082011 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:01.081969    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea7b308_e2e8_4aa4_9fd9_a86597cd0710.slice/crio-6f787732b06e101fea2a4c53a3b1ee316a99bd403a9c0f47cb951edd2e1c532f WatchSource:0}: Error finding container 6f787732b06e101fea2a4c53a3b1ee316a99bd403a9c0f47cb951edd2e1c532f: Status 404 returned error can't find the container with id 6f787732b06e101fea2a4c53a3b1ee316a99bd403a9c0f47cb951edd2e1c532f
Oct 30 17:30:01.090215 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:01.090187    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8f2fb8_fd66_45dc_918b_41b389b58436.slice/crio-106339885b6e8574a26b7677be9998298ba3ea6eedb2a804608ef0e8e9ac7750 WatchSource:0}: Error finding container 106339885b6e8574a26b7677be9998298ba3ea6eedb2a804608ef0e8e9ac7750: Status 404 returned error can't find the container with id 106339885b6e8574a26b7677be9998298ba3ea6eedb2a804608ef0e8e9ac7750
Oct 30 17:30:01.091805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.091778    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"]
Oct 30 17:30:01.143727 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:01.143698    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe915ae7_bed7_459b_9a8f_7e46afbbdc1e.slice/crio-02879ca2354627d169767bb03b2410b209642b22742f7c07ec35920b5a17291a WatchSource:0}: Error finding container 02879ca2354627d169767bb03b2410b209642b22742f7c07ec35920b5a17291a: Status 404 returned error can't find the container with id 02879ca2354627d169767bb03b2410b209642b22742f7c07ec35920b5a17291a
Oct 30 17:30:01.144244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.144208    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"]
Oct 30 17:30:01.187917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.187882    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"]
Oct 30 17:30:01.257773 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:01.257734    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475c528b_7bdb_4932_9453_2923fa9199b1.slice/crio-3f711dd1d16fd682d5a15e4baf92092b721239a5baeb8ade49350fa92ddc2852 WatchSource:0}: Error finding container 3f711dd1d16fd682d5a15e4baf92092b721239a5baeb8ade49350fa92ddc2852: Status 404 returned error can't find the container with id 3f711dd1d16fd682d5a15e4baf92092b721239a5baeb8ade49350fa92ddc2852
Oct 30 17:30:01.448325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.448264    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerStarted","Data":"6fbf945ed23605d366ddb30eb5c98d7448ca95e9c7312997a84d5640be6b9240"}
Oct 30 17:30:01.451679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.449994    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerStarted","Data":"106339885b6e8574a26b7677be9998298ba3ea6eedb2a804608ef0e8e9ac7750"}
Oct 30 17:30:01.451679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.451533    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerStarted","Data":"6f787732b06e101fea2a4c53a3b1ee316a99bd403a9c0f47cb951edd2e1c532f"}
Oct 30 17:30:01.458676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.453032    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerStarted","Data":"02879ca2354627d169767bb03b2410b209642b22742f7c07ec35920b5a17291a"}
Oct 30 17:30:01.465678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:01.461816    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerStarted","Data":"3f711dd1d16fd682d5a15e4baf92092b721239a5baeb8ade49350fa92ddc2852"}
Oct 30 17:30:02.474957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.473732    5534 generic.go:356] "Generic (PLEG): container finished" podID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerID="1a8c449974e0fcd49d8383e1eb90f2d89c149a5990f46747bfe2625a2be6c735" exitCode=60
Oct 30 17:30:02.474957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.473824    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerDied","Data":"1a8c449974e0fcd49d8383e1eb90f2d89c149a5990f46747bfe2625a2be6c735"}
Oct 30 17:30:02.474957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.474249    5534 scope.go:122] "RemoveContainer" containerID="1a8c449974e0fcd49d8383e1eb90f2d89c149a5990f46747bfe2625a2be6c735"
Oct 30 17:30:02.481555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.481512    5534 generic.go:356] "Generic (PLEG): container finished" podID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerID="f8f1db8300f151cb697e9d6eee92151ec9141551142a2dc4a1d3a55394d13605" exitCode=60
Oct 30 17:30:02.481736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.481718    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerDied","Data":"f8f1db8300f151cb697e9d6eee92151ec9141551142a2dc4a1d3a55394d13605"}
Oct 30 17:30:02.482103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.482084    5534 scope.go:122] "RemoveContainer" containerID="f8f1db8300f151cb697e9d6eee92151ec9141551142a2dc4a1d3a55394d13605"
Oct 30 17:30:02.485843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.485816    5534 generic.go:356] "Generic (PLEG): container finished" podID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerID="ae0114fce9e735c12dfdd8247f3d6e16fb771b237828eb61ba7961ba600034bf" exitCode=60
Oct 30 17:30:02.485946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.485930    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerDied","Data":"ae0114fce9e735c12dfdd8247f3d6e16fb771b237828eb61ba7961ba600034bf"}
Oct 30 17:30:02.486294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.486273    5534 scope.go:122] "RemoveContainer" containerID="ae0114fce9e735c12dfdd8247f3d6e16fb771b237828eb61ba7961ba600034bf"
Oct 30 17:30:02.490125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.488595    5534 generic.go:356] "Generic (PLEG): container finished" podID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerID="61932f09d01e193d03655a13c7c1200ac9dd1f375fd384dca5c0b026a710c801" exitCode=60
Oct 30 17:30:02.490125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.488714    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerDied","Data":"61932f09d01e193d03655a13c7c1200ac9dd1f375fd384dca5c0b026a710c801"}
Oct 30 17:30:02.490125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.489032    5534 scope.go:122] "RemoveContainer" containerID="61932f09d01e193d03655a13c7c1200ac9dd1f375fd384dca5c0b026a710c801"
Oct 30 17:30:02.498217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.498183    5534 generic.go:356] "Generic (PLEG): container finished" podID="475c528b-7bdb-4932-9453-2923fa9199b1" containerID="c36671b2111b43c0939224023e371fd391739c4002e445128be730987e739dc4" exitCode=60
Oct 30 17:30:02.498387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.498329    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerDied","Data":"c36671b2111b43c0939224023e371fd391739c4002e445128be730987e739dc4"}
Oct 30 17:30:02.498487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.498447    5534 scope.go:122] "RemoveContainer" containerID="c36671b2111b43c0939224023e371fd391739c4002e445128be730987e739dc4"
Oct 30 17:30:02.723954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:02.723842    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:03.508778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.508735    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerStarted","Data":"2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c"}
Oct 30 17:30:03.510948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.510922    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerStarted","Data":"001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869"}
Oct 30 17:30:03.513136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.513105    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerStarted","Data":"ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c"}
Oct 30 17:30:03.515161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.515139    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerStarted","Data":"9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c"}
Oct 30 17:30:03.517281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.517258    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerStarted","Data":"fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753"}
Oct 30 17:30:03.552027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.551971    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" podStartSLOduration=3.551954269 podStartE2EDuration="3.551954269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.549086604 +0000 UTC m=+1009.311130429" watchObservedRunningTime="2026-10-30 17:30:03.551954269 +0000 UTC m=+1009.313998126"
Oct 30 17:30:03.590504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.590451    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" podStartSLOduration=3.590437747 podStartE2EDuration="3.590437747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.589491378 +0000 UTC m=+1009.351535193" watchObservedRunningTime="2026-10-30 17:30:03.590437747 +0000 UTC m=+1009.352481580"
Oct 30 17:30:03.629345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.629285    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" podStartSLOduration=3.629270433 podStartE2EDuration="3.629270433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.628485527 +0000 UTC m=+1009.390529342" watchObservedRunningTime="2026-10-30 17:30:03.629270433 +0000 UTC m=+1009.391314246"
Oct 30 17:30:03.670606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.670557    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" podStartSLOduration=3.6705373100000003 podStartE2EDuration="3.67053731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.66821296 +0000 UTC m=+1009.430256773" watchObservedRunningTime="2026-10-30 17:30:03.67053731 +0000 UTC m=+1009.432581165"
Oct 30 17:30:03.708025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.707953    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" podStartSLOduration=3.707933768 podStartE2EDuration="3.707933768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.70730535 +0000 UTC m=+1009.469349164" watchObservedRunningTime="2026-10-30 17:30:03.707933768 +0000 UTC m=+1009.469977581"
Oct 30 17:30:03.999602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:03.999549    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:04.526735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.526693    5534 generic.go:356] "Generic (PLEG): container finished" podID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerID="2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c" exitCode=60
Oct 30 17:30:04.527221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.526783    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerDied","Data":"2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c"}
Oct 30 17:30:04.527221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.526847    5534 scope.go:122] "RemoveContainer" containerID="1a8c449974e0fcd49d8383e1eb90f2d89c149a5990f46747bfe2625a2be6c735"
Oct 30 17:30:04.527221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.527092    5534 scope.go:122] "RemoveContainer" containerID="2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c"
Oct 30 17:30:04.527483 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:04.527455    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn_odh-ai-gateway-infra(bff4dbed-a05b-42f9-8635-3e818aef5c98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98"
Oct 30 17:30:04.528929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.528903    5534 generic.go:356] "Generic (PLEG): container finished" podID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerID="001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869" exitCode=60
Oct 30 17:30:04.529066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.528984    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerDied","Data":"001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869"}
Oct 30 17:30:04.529225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.529211    5534 scope.go:122] "RemoveContainer" containerID="001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869"
Oct 30 17:30:04.529494 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:04.529474    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-fvjhp_odh-ai-gateway-infra(7a8f2fb8-fd66-45dc-918b-41b389b58436)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436"
Oct 30 17:30:04.531044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.531014    5534 generic.go:356] "Generic (PLEG): container finished" podID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerID="ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c" exitCode=60
Oct 30 17:30:04.531143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.531049    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerDied","Data":"ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c"}
Oct 30 17:30:04.531354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.531339    5534 scope.go:122] "RemoveContainer" containerID="ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c"
Oct 30 17:30:04.531660 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:04.531621    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z_odh-ai-gateway-infra(1ea7b308-e2e8-4aa4-9fd9-a86597cd0710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710"
Oct 30 17:30:04.533301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.533282    5534 generic.go:356] "Generic (PLEG): container finished" podID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerID="9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c" exitCode=60
Oct 30 17:30:04.533414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.533387    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerDied","Data":"9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c"}
Oct 30 17:30:04.533739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.533718    5534 scope.go:122] "RemoveContainer" containerID="9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c"
Oct 30 17:30:04.533997 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:04.533979    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm_odh-ai-gateway-infra(fe915ae7-bed7-459b-9a8f-7e46afbbdc1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e"
Oct 30 17:30:04.535679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.535619    5534 generic.go:356] "Generic (PLEG): container finished" podID="475c528b-7bdb-4932-9453-2923fa9199b1" containerID="fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753" exitCode=60
Oct 30 17:30:04.535748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.535693    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerDied","Data":"fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753"}
Oct 30 17:30:04.535922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.535905    5534 scope.go:122] "RemoveContainer" containerID="fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753"
Oct 30 17:30:04.536128 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:04.536112    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb_odh-ai-gateway-infra(475c528b-7bdb-4932-9453-2923fa9199b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" podUID="475c528b-7bdb-4932-9453-2923fa9199b1"
Oct 30 17:30:04.540356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.540339    5534 scope.go:122] "RemoveContainer" containerID="f8f1db8300f151cb697e9d6eee92151ec9141551142a2dc4a1d3a55394d13605"
Oct 30 17:30:04.553715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.553691    5534 scope.go:122] "RemoveContainer" containerID="ae0114fce9e735c12dfdd8247f3d6e16fb771b237828eb61ba7961ba600034bf"
Oct 30 17:30:04.567482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.567443    5534 scope.go:122] "RemoveContainer" containerID="61932f09d01e193d03655a13c7c1200ac9dd1f375fd384dca5c0b026a710c801"
Oct 30 17:30:04.581090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.581053    5534 scope.go:122] "RemoveContainer" containerID="c36671b2111b43c0939224023e371fd391739c4002e445128be730987e739dc4"
Oct 30 17:30:04.788859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:04.788742    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:05.545357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:05.545326    5534 scope.go:122] "RemoveContainer" containerID="ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c"
Oct 30 17:30:05.545825 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:05.545728    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z_odh-ai-gateway-infra(1ea7b308-e2e8-4aa4-9fd9-a86597cd0710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710"
Oct 30 17:30:05.547429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:05.547403    5534 scope.go:122] "RemoveContainer" containerID="9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c"
Oct 30 17:30:05.547790 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:05.547765    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm_odh-ai-gateway-infra(fe915ae7-bed7-459b-9a8f-7e46afbbdc1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e"
Oct 30 17:30:05.549416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:05.549394    5534 scope.go:122] "RemoveContainer" containerID="fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753"
Oct 30 17:30:05.549724 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:05.549698    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb_odh-ai-gateway-infra(475c528b-7bdb-4932-9453-2923fa9199b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" podUID="475c528b-7bdb-4932-9453-2923fa9199b1"
Oct 30 17:30:05.551189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:05.551172    5534 scope.go:122] "RemoveContainer" containerID="2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c"
Oct 30 17:30:05.551484 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:05.551461    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn_odh-ai-gateway-infra(bff4dbed-a05b-42f9-8635-3e818aef5c98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98"
Oct 30 17:30:05.552946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:05.552929    5534 scope.go:122] "RemoveContainer" containerID="001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869"
Oct 30 17:30:05.553142 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:05.553126    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-fvjhp_odh-ai-gateway-infra(7a8f2fb8-fd66-45dc-918b-41b389b58436)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436"
Oct 30 17:30:05.997858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:05.997816    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"]
Oct 30 17:30:06.004570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.004544    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.020027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.019997    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"openshift-service-ca.crt\""
Oct 30 17:30:06.020182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.020044    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"default-dockercfg-jgc4t\""
Oct 30 17:30:06.020182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.020126    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"kube-root-ca.crt\""
Oct 30 17:30:06.023417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.023390    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"]
Oct 30 17:30:06.032057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.032024    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"distinct-w4-0e7403-kserve-self-signed-certs\""
Oct 30 17:30:06.053260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.053226    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:06.108784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.108744    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.108980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.108802    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.109020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.108982    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.109068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.109054    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69hq\" (UniqueName: \"kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.109161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.109143    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.109217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.109200    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.212926    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.213043    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69hq\" (UniqueName: \"kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.213110    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.213184    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.213238    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.213273    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.215395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.214220    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.229232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.224190    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.229232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.224482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.229232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.225285    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.234677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.231932    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.250240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.250161    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69hq\" (UniqueName: \"kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq\") pod \"distinct-w4-0e7403-kserve-588c78b8d8-kjzlp\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.321723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.321687    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:06.518849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.518761    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:06.637969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:06.637937    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"]
Oct 30 17:30:06.638580 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:06.638556    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c768139_2889_4936_b9bd_34fd5d0718fd.slice/crio-82d74f314dd0f7df352beab1b9081df6cbf1f13820ab0011ac70f5df015779da WatchSource:0}: Error finding container 82d74f314dd0f7df352beab1b9081df6cbf1f13820ab0011ac70f5df015779da: Status 404 returned error can't find the container with id 82d74f314dd0f7df352beab1b9081df6cbf1f13820ab0011ac70f5df015779da
Oct 30 17:30:07.573753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.573715    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" event={"ID":"4c768139-2889-4936-b9bd-34fd5d0718fd","Type":"ContainerStarted","Data":"0a9047705dedf7f24614e7335288740731f8a31a7929e551a26c8acea79e1db9"}
Oct 30 17:30:07.573960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.573770    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" event={"ID":"4c768139-2889-4936-b9bd-34fd5d0718fd","Type":"ContainerStarted","Data":"82d74f314dd0f7df352beab1b9081df6cbf1f13820ab0011ac70f5df015779da"}
Oct 30 17:30:07.617685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.617611    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podStartSLOduration=2.617593582 podStartE2EDuration="2.617593582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:07.614962201 +0000 UTC m=+1013.377006028" watchObservedRunningTime="2026-10-30 17:30:07.617593582 +0000 UTC m=+1013.379637394"
Oct 30 17:30:07.692018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.691978    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"]
Oct 30 17:30:07.696833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.696798    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.729279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.729244    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"]
Oct 30 17:30:07.847205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847173    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847237    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847379    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847531    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847677    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847747    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847794    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847818    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.847885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.847846    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfrz\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.948950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.948911    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.948983    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949030    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949099    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949154    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949192    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949213    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949246    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949418    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfrz\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949355    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949512    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949578    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.949779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.949754    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.950035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.950006    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.951869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.951848    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.951998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.951980    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.973250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.973207    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:07.973398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:07.973266    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfrz\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:08.020663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.020606    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:08.322064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.321986    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:08.351138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.351111    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"]
Oct 30 17:30:08.351696 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:08.351671    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c282ed_89ac_48f6_9f4e_7946576d5dd0.slice/crio-22b196ece042291927ff789974389a0eb59a03c5dfc952a3f1819ea5fd23ce4b WatchSource:0}: Error finding container 22b196ece042291927ff789974389a0eb59a03c5dfc952a3f1819ea5fd23ce4b: Status 404 returned error can't find the container with id 22b196ece042291927ff789974389a0eb59a03c5dfc952a3f1819ea5fd23ce4b
Oct 30 17:30:08.354509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.354473    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:08.354602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.354564    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:08.354602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.354596    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:08.424254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.424215    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:08.585161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.585062    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" event={"ID":"e9c282ed-89ac-48f6-9f4e-7946576d5dd0","Type":"ContainerStarted","Data":"6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d"}
Oct 30 17:30:08.585161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.585131    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" event={"ID":"e9c282ed-89ac-48f6-9f4e-7946576d5dd0","Type":"ContainerStarted","Data":"22b196ece042291927ff789974389a0eb59a03c5dfc952a3f1819ea5fd23ce4b"}
Oct 30 17:30:08.632356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:08.632299    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podStartSLOduration=1.63228314 podStartE2EDuration="1.63228314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:08.629315685 +0000 UTC m=+1014.391359500" watchObservedRunningTime="2026-10-30 17:30:08.63228314 +0000 UTC m=+1014.394326953"
Oct 30 17:30:09.020800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:09.020766    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:09.022748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:09.022725    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:09.022863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:09.022777    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:10.021315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:10.021279    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:10.021720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:10.021331    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:10.109948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:10.109911    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:11.021449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:11.021398    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:11.021967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:11.021465    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:11.914852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:11.914810    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:12.021428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:12.021390    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:12.021587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:12.021452    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:13.021947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:13.021911    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:13.022477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:13.022004    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:13.698369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:13.698330    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:14.022409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:14.022320    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:14.022409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:14.022386    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:15.021610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.021567    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 17:30:15.021811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.021723    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 17:30:15.325080 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:15.324964    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:30:15.369731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.369703    5534 scope.go:122] "RemoveContainer" containerID="ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c"
Oct 30 17:30:15.371077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.371047    5534 scope.go:122] "RemoveContainer" containerID="fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753"
Oct 30 17:30:15.372441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.372041    5534 scope.go:122] "RemoveContainer" containerID="9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c"
Oct 30 17:30:15.372740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.372714    5534 scope.go:122] "RemoveContainer" containerID="2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c"
Oct 30 17:30:15.654685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:15.654450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:16.028054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.027129    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:16.028054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.027875    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:16.029355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.029329    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:30:16.349737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.349709    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:16.436530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.436102    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:16.825669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.818165    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerStarted","Data":"7dc285d942f06e9b35454d13f78d9b91244428555fb1d20928b5248cf7f63125"}
Oct 30 17:30:16.856673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.844757    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerStarted","Data":"f754b018b26a12f8eeb61c7d6fc678c78d490a3ab6205d5960e7f755fb79748c"}
Oct 30 17:30:16.856673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.851773    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerStarted","Data":"2eeed038065a47e3f5e3c4851882958e262c357dc1e0d40535c12645dbf57987"}
Oct 30 17:30:16.856673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.852544    5534 scope.go:122] "RemoveContainer" containerID="2eeed038065a47e3f5e3c4851882958e262c357dc1e0d40535c12645dbf57987"
Oct 30 17:30:16.856673 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:16.853109    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb_odh-ai-gateway-infra(475c528b-7bdb-4932-9453-2923fa9199b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" podUID="475c528b-7bdb-4932-9453-2923fa9199b1"
Oct 30 17:30:16.898683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.878629    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerStarted","Data":"d7621492a4c9bd4093b36541d70ed12c4d1f3083c580b0a45f8b2daeacfbb834"}
Oct 30 17:30:16.898683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:16.879061    5534 scope.go:122] "RemoveContainer" containerID="d7621492a4c9bd4093b36541d70ed12c4d1f3083c580b0a45f8b2daeacfbb834"
Oct 30 17:30:16.911179 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:16.910671    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn_odh-ai-gateway-infra(bff4dbed-a05b-42f9-8635-3e818aef5c98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98"
Oct 30 17:30:17.025669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.025536    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:30:17.815823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.815787    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:17.879274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.879234    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"]
Oct 30 17:30:17.907183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.907143    5534 generic.go:356] "Generic (PLEG): container finished" podID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerID="7dc285d942f06e9b35454d13f78d9b91244428555fb1d20928b5248cf7f63125" exitCode=60
Oct 30 17:30:17.907389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.907287    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerDied","Data":"7dc285d942f06e9b35454d13f78d9b91244428555fb1d20928b5248cf7f63125"}
Oct 30 17:30:17.907389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.907351    5534 scope.go:122] "RemoveContainer" containerID="ca42f3f5c465c8179a7ccb05fc3a13df612cac3e47595d55ff13c3eb60b1397c"
Oct 30 17:30:17.910455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.910425    5534 generic.go:356] "Generic (PLEG): container finished" podID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerID="f754b018b26a12f8eeb61c7d6fc678c78d490a3ab6205d5960e7f755fb79748c" exitCode=60
Oct 30 17:30:17.910578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.910494    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerDied","Data":"f754b018b26a12f8eeb61c7d6fc678c78d490a3ab6205d5960e7f755fb79748c"}
Oct 30 17:30:17.911076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.911057    5534 scope.go:122] "RemoveContainer" containerID="f754b018b26a12f8eeb61c7d6fc678c78d490a3ab6205d5960e7f755fb79748c"
Oct 30 17:30:17.911440 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:17.911417    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm_odh-ai-gateway-infra(fe915ae7-bed7-459b-9a8f-7e46afbbdc1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e"
Oct 30 17:30:17.913034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.913011    5534 generic.go:356] "Generic (PLEG): container finished" podID="475c528b-7bdb-4932-9453-2923fa9199b1" containerID="2eeed038065a47e3f5e3c4851882958e262c357dc1e0d40535c12645dbf57987" exitCode=60
Oct 30 17:30:17.913138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.913078    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerDied","Data":"2eeed038065a47e3f5e3c4851882958e262c357dc1e0d40535c12645dbf57987"}
Oct 30 17:30:17.913531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.913514    5534 scope.go:122] "RemoveContainer" containerID="2eeed038065a47e3f5e3c4851882958e262c357dc1e0d40535c12645dbf57987"
Oct 30 17:30:17.913899 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:17.913877    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb_odh-ai-gateway-infra(475c528b-7bdb-4932-9453-2923fa9199b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" podUID="475c528b-7bdb-4932-9453-2923fa9199b1"
Oct 30 17:30:17.915584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.915564    5534 generic.go:356] "Generic (PLEG): container finished" podID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerID="d7621492a4c9bd4093b36541d70ed12c4d1f3083c580b0a45f8b2daeacfbb834" exitCode=60
Oct 30 17:30:17.915733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.915632    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerDied","Data":"d7621492a4c9bd4093b36541d70ed12c4d1f3083c580b0a45f8b2daeacfbb834"}
Oct 30 17:30:17.916231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.916213    5534 scope.go:122] "RemoveContainer" containerID="d7621492a4c9bd4093b36541d70ed12c4d1f3083c580b0a45f8b2daeacfbb834"
Oct 30 17:30:17.916446 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:17.916425    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn_odh-ai-gateway-infra(bff4dbed-a05b-42f9-8635-3e818aef5c98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98"
Oct 30 17:30:17.923862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.923837    5534 scope.go:122] "RemoveContainer" containerID="9d102bf7b26c6f88bf7596d81f1336d408fde9e1907635a3a8f806af62a10e8c"
Oct 30 17:30:17.957898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.957830    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"]
Oct 30 17:30:17.961469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.961345    5534 scope.go:122] "RemoveContainer" containerID="fd514f1a920443d153fe887fe8fb0364fbc7c4933fd0223f0b782a9969429753"
Oct 30 17:30:17.980359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.980313    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"]
Oct 30 17:30:17.983932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:17.983904    5534 scope.go:122] "RemoveContainer" containerID="2163a7df4856016c0cd4fd842cb53b0236e6d889fd127f102df8521f5180831c"
Oct 30 17:30:18.065685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.065628    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"]
Oct 30 17:30:18.105790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.105764    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:18.231851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.231801    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb\" (UniqueName: \"kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb\") pod \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") "
Oct 30 17:30:18.232027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.231866    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca\") pod \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\" (UID: \"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710\") "
Oct 30 17:30:18.232288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.232258    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca" pod "1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" (UID: "1ea7b308-e2e8-4aa4-9fd9-a86597cd0710"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:18.232563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.232548    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:18.234209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.234184    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb" pod "1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" (UID: "1ea7b308-e2e8-4aa4-9fd9-a86597cd0710"). InnerVolumeSpecName "kube-api-access-6hffb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:18.333561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.333478    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hffb\" (UniqueName: \"kubernetes.io/projected/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710-kube-api-access-6hffb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:18.363248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.363209    5534 scope.go:122] "RemoveContainer" containerID="001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869"
Oct 30 17:30:18.924114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.924025    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"
Oct 30 17:30:18.924114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.924064    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z" event={"ID":"1ea7b308-e2e8-4aa4-9fd9-a86597cd0710","Type":"ContainerDied","Data":"6f787732b06e101fea2a4c53a3b1ee316a99bd403a9c0f47cb951edd2e1c532f"}
Oct 30 17:30:18.924671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.924134    5534 scope.go:122] "RemoveContainer" containerID="7dc285d942f06e9b35454d13f78d9b91244428555fb1d20928b5248cf7f63125"
Oct 30 17:30:18.933725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:18.933690    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerStarted","Data":"5d25aea557eeef7a093f006bf873db89138277a5a631c576bb4c4c50b6295f29"}
Oct 30 17:30:19.032615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.032474    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"]
Oct 30 17:30:19.038575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.038368    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889690-4ph6z"]
Oct 30 17:30:19.124661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.123671    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:19.188462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.188398    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:19.211466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.211439    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:19.247900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.247859    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca\") pod \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") "
Oct 30 17:30:19.248097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.248073    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8\" (UniqueName: \"kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8\") pod \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\" (UID: \"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e\") "
Oct 30 17:30:19.248259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.248234    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca" pod "fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" (UID: "fe915ae7-bed7-459b-9a8f-7e46afbbdc1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.248539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.248514    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.250867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.250839    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8" pod "fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" (UID: "fe915ae7-bed7-459b-9a8f-7e46afbbdc1e"). InnerVolumeSpecName "kube-api-access-jzgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.349583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.349550    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6\" (UniqueName: \"kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6\") pod \"475c528b-7bdb-4932-9453-2923fa9199b1\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") "
Oct 30 17:30:19.349777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.349656    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca\") pod \"bff4dbed-a05b-42f9-8635-3e818aef5c98\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") "
Oct 30 17:30:19.349777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.349720    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca\") pod \"475c528b-7bdb-4932-9453-2923fa9199b1\" (UID: \"475c528b-7bdb-4932-9453-2923fa9199b1\") "
Oct 30 17:30:19.349865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.349850    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b\" (UniqueName: \"kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b\") pod \"bff4dbed-a05b-42f9-8635-3e818aef5c98\" (UID: \"bff4dbed-a05b-42f9-8635-3e818aef5c98\") "
Oct 30 17:30:19.349993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.349966    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca" pod "bff4dbed-a05b-42f9-8635-3e818aef5c98" (UID: "bff4dbed-a05b-42f9-8635-3e818aef5c98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.350079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.350060    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca" pod "475c528b-7bdb-4932-9453-2923fa9199b1" (UID: "475c528b-7bdb-4932-9453-2923fa9199b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.350309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.350288    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzgk8\" (UniqueName: \"kubernetes.io/projected/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e-kube-api-access-jzgk8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.350309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.350308    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff4dbed-a05b-42f9-8635-3e818aef5c98-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.350695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.350320    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475c528b-7bdb-4932-9453-2923fa9199b1-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.351919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.351899    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6" pod "475c528b-7bdb-4932-9453-2923fa9199b1" (UID: "475c528b-7bdb-4932-9453-2923fa9199b1"). InnerVolumeSpecName "kube-api-access-fjbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.352007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.351976    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b" pod "bff4dbed-a05b-42f9-8635-3e818aef5c98" (UID: "bff4dbed-a05b-42f9-8635-3e818aef5c98"). InnerVolumeSpecName "kube-api-access-z5f2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.451481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.451393    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5f2b\" (UniqueName: \"kubernetes.io/projected/bff4dbed-a05b-42f9-8635-3e818aef5c98-kube-api-access-z5f2b\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.451481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.451427    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbm6\" (UniqueName: \"kubernetes.io/projected/475c528b-7bdb-4932-9453-2923fa9199b1-kube-api-access-fjbm6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.518583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.518549    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:19.942701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.942658    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm" event={"ID":"fe915ae7-bed7-459b-9a8f-7e46afbbdc1e","Type":"ContainerDied","Data":"02879ca2354627d169767bb03b2410b209642b22742f7c07ec35920b5a17291a"}
Oct 30 17:30:19.942701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.942677    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"
Oct 30 17:30:19.943245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.942734    5534 scope.go:122] "RemoveContainer" containerID="f754b018b26a12f8eeb61c7d6fc678c78d490a3ab6205d5960e7f755fb79748c"
Oct 30 17:30:19.944299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.944281    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"
Oct 30 17:30:19.944420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.944344    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb" event={"ID":"475c528b-7bdb-4932-9453-2923fa9199b1","Type":"ContainerDied","Data":"3f711dd1d16fd682d5a15e4baf92092b721239a5baeb8ade49350fa92ddc2852"}
Oct 30 17:30:19.946091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.946058    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"
Oct 30 17:30:19.946210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.946056    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn" event={"ID":"bff4dbed-a05b-42f9-8635-3e818aef5c98","Type":"ContainerDied","Data":"6fbf945ed23605d366ddb30eb5c98d7448ca95e9c7312997a84d5640be6b9240"}
Oct 30 17:30:19.948755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.948729    5534 generic.go:356] "Generic (PLEG): container finished" podID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerID="5d25aea557eeef7a093f006bf873db89138277a5a631c576bb4c4c50b6295f29" exitCode=60
Oct 30 17:30:19.948849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.948834    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerDied","Data":"5d25aea557eeef7a093f006bf873db89138277a5a631c576bb4c4c50b6295f29"}
Oct 30 17:30:19.949242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.949225    5534 scope.go:122] "RemoveContainer" containerID="5d25aea557eeef7a093f006bf873db89138277a5a631c576bb4c4c50b6295f29"
Oct 30 17:30:19.949518 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:19.949495    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-fvjhp_odh-ai-gateway-infra(7a8f2fb8-fd66-45dc-918b-41b389b58436)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436"
Oct 30 17:30:19.962446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.962424    5534 scope.go:122] "RemoveContainer" containerID="2eeed038065a47e3f5e3c4851882958e262c357dc1e0d40535c12645dbf57987"
Oct 30 17:30:19.975093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.975071    5534 scope.go:122] "RemoveContainer" containerID="d7621492a4c9bd4093b36541d70ed12c4d1f3083c580b0a45f8b2daeacfbb834"
Oct 30 17:30:19.984485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:19.984466    5534 scope.go:122] "RemoveContainer" containerID="001bd06122f728c851daf4d80dbf97dddae8a0d371905c84172a3ebf44437869"
Oct 30 17:30:20.075675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.075615    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"]
Oct 30 17:30:20.081110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.081078    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889690-tjtmn"]
Oct 30 17:30:20.129105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.129051    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"]
Oct 30 17:30:20.132451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.132421    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889690-bk2gm"]
Oct 30 17:30:20.139076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.139046    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"]
Oct 30 17:30:20.168380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.168332    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"]
Oct 30 17:30:20.173062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.173031    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889690-bfbcb"]
Oct 30 17:30:20.375184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.375135    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" path="/var/lib/kubelet/pods/1ea7b308-e2e8-4aa4-9fd9-a86597cd0710/volumes"
Oct 30 17:30:20.375803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.375782    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" path="/var/lib/kubelet/pods/475c528b-7bdb-4932-9453-2923fa9199b1/volumes"
Oct 30 17:30:20.376357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.376339    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" path="/var/lib/kubelet/pods/bff4dbed-a05b-42f9-8635-3e818aef5c98/volumes"
Oct 30 17:30:20.377305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:20.377284    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" path="/var/lib/kubelet/pods/fe915ae7-bed7-459b-9a8f-7e46afbbdc1e/volumes"
Oct 30 17:30:21.125897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.125865    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:21.225162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.225078    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"]
Oct 30 17:30:21.225977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.225952    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.225980    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.225995    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226003    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226033    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226041    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226058    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.226066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226066    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226083    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226090    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226099    5534 cpu_manager.go:414] "containerMap: removing container" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226105    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226114    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226121    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226142    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226149    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226157    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226164    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226176    5534 cpu_manager.go:414] "containerMap: removing container" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226182    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226204    5534 cpu_manager.go:414] "containerMap: removing container" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226211    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226237    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226244    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226372    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226385    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226393    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226407    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226416    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226427    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226437    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.226442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226449    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.227761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226459    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.227761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226468    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.227761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226478    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe915ae7-bed7-459b-9a8f-7e46afbbdc1e" containerName="cleanup"
Oct 30 17:30:21.227761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.226488    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.230622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.230597    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.248554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.245499    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:30:21.248976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.248952    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"]
Oct 30 17:30:21.280258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.280217    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca\") pod \"7a8f2fb8-fd66-45dc-918b-41b389b58436\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") "
Oct 30 17:30:21.280448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.280361    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm\" (UniqueName: \"kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm\") pod \"7a8f2fb8-fd66-45dc-918b-41b389b58436\" (UID: \"7a8f2fb8-fd66-45dc-918b-41b389b58436\") "
Oct 30 17:30:21.280623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.280601    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca" pod "7a8f2fb8-fd66-45dc-918b-41b389b58436" (UID: "7a8f2fb8-fd66-45dc-918b-41b389b58436"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:21.280982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.280962    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f2fb8-fd66-45dc-918b-41b389b58436-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.285670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.283251    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm" pod "7a8f2fb8-fd66-45dc-918b-41b389b58436" (UID: "7a8f2fb8-fd66-45dc-918b-41b389b58436"). InnerVolumeSpecName "kube-api-access-jfwrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:21.382418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.382374    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj27l\" (UniqueName: \"kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.382580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.382556    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.382761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.382740    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.382899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.382878    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwrm\" (UniqueName: \"kubernetes.io/projected/7a8f2fb8-fd66-45dc-918b-41b389b58436-kube-api-access-jfwrm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.456228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.456183    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"]
Oct 30 17:30:21.457049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457022    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.457049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457049    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff4dbed-a05b-42f9-8635-3e818aef5c98" containerName="cleanup"
Oct 30 17:30:21.457241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457073    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.457241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457081    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.457241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457139    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.457241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457148    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.457436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457342    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea7b308-e2e8-4aa4-9fd9-a86597cd0710" containerName="cleanup"
Oct 30 17:30:21.457436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457358    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="475c528b-7bdb-4932-9453-2923fa9199b1" containerName="cleanup"
Oct 30 17:30:21.457436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.457379    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" containerName="cleanup"
Oct 30 17:30:21.469059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.468240    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.483985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.483901    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"]
Oct 30 17:30:21.484117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.484001    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.484166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.484010    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"distinct-2-w5-db98b0-kserve-self-signed-certs\""
Oct 30 17:30:21.484201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.484168    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.485691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.484288    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj27l\" (UniqueName: \"kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.485691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.485051    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.487815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.487589    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.508702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.508666    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj27l\" (UniqueName: \"kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l\") pod \"maas-api-revoke-sub-fed5fc69-9m8p4\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.524935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.524895    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:21.545243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.545208    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:21.586331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.586283    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.586545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.586477    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.586689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.586611    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.586860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.586833    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgq74\" (UniqueName: \"kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.586982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.586941    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.587043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.587002    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.680441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.680118    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:21.690527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.690458    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgq74\" (UniqueName: \"kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.691561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.691517    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.691798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.691733    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.692044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.691965    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.692281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.692227    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.692527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.692472    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.696325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.693178    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.696325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.693557    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.697448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.697379    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.721656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.720582    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.723194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.723119    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgq74\" (UniqueName: \"kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.724502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.724444    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs\") pod \"distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.811745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.811600    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:21.862116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.862080    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"]
Oct 30 17:30:21.862596 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:21.862566    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38b74e4_74b9_49e3_a58c_15d004db0cb2.slice/crio-9fa7e5eced60a2b970d99d9482a639c9b267e57426e79fcfb2b32f4223b2ae34 WatchSource:0}: Error finding container 9fa7e5eced60a2b970d99d9482a639c9b267e57426e79fcfb2b32f4223b2ae34: Status 404 returned error can't find the container with id 9fa7e5eced60a2b970d99d9482a639c9b267e57426e79fcfb2b32f4223b2ae34
Oct 30 17:30:21.973002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.972974    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"
Oct 30 17:30:21.973199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.973014    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp" event={"ID":"7a8f2fb8-fd66-45dc-918b-41b389b58436","Type":"ContainerDied","Data":"106339885b6e8574a26b7677be9998298ba3ea6eedb2a804608ef0e8e9ac7750"}
Oct 30 17:30:21.973199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.973086    5534 scope.go:122] "RemoveContainer" containerID="5d25aea557eeef7a093f006bf873db89138277a5a631c576bb4c4c50b6295f29"
Oct 30 17:30:21.974855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:21.974826    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4" event={"ID":"e38b74e4-74b9-49e3-a58c-15d004db0cb2","Type":"ContainerStarted","Data":"9fa7e5eced60a2b970d99d9482a639c9b267e57426e79fcfb2b32f4223b2ae34"}
Oct 30 17:30:22.027803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.027756    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"]
Oct 30 17:30:22.032968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.032926    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-fvjhp"]
Oct 30 17:30:22.124656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.124610    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"]
Oct 30 17:30:22.124838 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:22.124817    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87554784_d0fe_4228_8dcf_c9a0af78a20f.slice/crio-b2ef065a18dee30e92f40943f60633f80dd248028a8d9127a9647b669c7aebd3 WatchSource:0}: Error finding container b2ef065a18dee30e92f40943f60633f80dd248028a8d9127a9647b669c7aebd3: Status 404 returned error can't find the container with id b2ef065a18dee30e92f40943f60633f80dd248028a8d9127a9647b669c7aebd3
Oct 30 17:30:22.377178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.373318    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8f2fb8-fd66-45dc-918b-41b389b58436" path="/var/lib/kubelet/pods/7a8f2fb8-fd66-45dc-918b-41b389b58436/volumes"
Oct 30 17:30:22.724926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.724838    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"]
Oct 30 17:30:22.729212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.729184    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.760837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.760800    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"]
Oct 30 17:30:22.818023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.817983    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818033    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818116    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818162    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818243    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818268    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmrl\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818287    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818372    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.818511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.818503    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919175    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919235    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919405    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919458    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919504    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919534    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919558    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919631    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919685    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919697    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmrl\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.919865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919760    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.920046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.919989    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.920511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.920489    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.920629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.920493    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.922511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.922486    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.922596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.922566    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.942962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.942922    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.943164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.943138    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmrl\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:22.984219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.984106    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" event={"ID":"87554784-d0fe-4228-8dcf-c9a0af78a20f","Type":"ContainerStarted","Data":"59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7"}
Oct 30 17:30:22.984219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.984158    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" event={"ID":"87554784-d0fe-4228-8dcf-c9a0af78a20f","Type":"ContainerStarted","Data":"b2ef065a18dee30e92f40943f60633f80dd248028a8d9127a9647b669c7aebd3"}
Oct 30 17:30:22.986086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.986058    5534 generic.go:356] "Generic (PLEG): container finished" podID="e38b74e4-74b9-49e3-a58c-15d004db0cb2" containerID="961234c102661e2c331d1ba4ac8648328b6d2c7d874d53f26860af43b3a72b67" exitCode=0
Oct 30 17:30:22.986274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:22.986146    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4" event={"ID":"e38b74e4-74b9-49e3-a58c-15d004db0cb2","Type":"ContainerDied","Data":"961234c102661e2c331d1ba4ac8648328b6d2c7d874d53f26860af43b3a72b67"}
Oct 30 17:30:23.026614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.026555    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podStartSLOduration=2.026537237 podStartE2EDuration="2.026537237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:23.024408381 +0000 UTC m=+1028.786452209" watchObservedRunningTime="2026-10-30 17:30:23.026537237 +0000 UTC m=+1028.788581049"
Oct 30 17:30:23.044104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.044066    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:23.375954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.375920    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"]
Oct 30 17:30:23.376612 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:23.376583    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd52897_715d_4d7b_b3ef_79aac18b7226.slice/crio-97115eb1b65ea11b173077a1b0bed6d012f72ab05ef1db8dc64ec01510ddd3b3 WatchSource:0}: Error finding container 97115eb1b65ea11b173077a1b0bed6d012f72ab05ef1db8dc64ec01510ddd3b3: Status 404 returned error can't find the container with id 97115eb1b65ea11b173077a1b0bed6d012f72ab05ef1db8dc64ec01510ddd3b3
Oct 30 17:30:23.379396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.379363    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:23.379737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.379433    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:23.379737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.379461    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:23.590859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.590828    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:23.812700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.812573    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:23.998845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.998802    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" event={"ID":"9cd52897-715d-4d7b-b3ef-79aac18b7226","Type":"ContainerStarted","Data":"ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2"}
Oct 30 17:30:23.999030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:23.998870    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" event={"ID":"9cd52897-715d-4d7b-b3ef-79aac18b7226","Type":"ContainerStarted","Data":"97115eb1b65ea11b173077a1b0bed6d012f72ab05ef1db8dc64ec01510ddd3b3"}
Oct 30 17:30:24.044785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.044746    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:24.046023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.045983    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:24.046147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.046044    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:24.059286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.059213    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podStartSLOduration=2.059191029 podStartE2EDuration="2.059191029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:24.058283166 +0000 UTC m=+1029.820327020" watchObservedRunningTime="2026-10-30 17:30:24.059191029 +0000 UTC m=+1029.821234844"
Oct 30 17:30:24.222265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.222234    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:24.248503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.248476    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca\") pod \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") "
Oct 30 17:30:24.248718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.248527    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l\" (UniqueName: \"kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l\") pod \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") "
Oct 30 17:30:24.248796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.248740    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca\") pod \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\" (UID: \"e38b74e4-74b9-49e3-a58c-15d004db0cb2\") "
Oct 30 17:30:24.248948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.248919    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca" pod "e38b74e4-74b9-49e3-a58c-15d004db0cb2" (UID: "e38b74e4-74b9-49e3-a58c-15d004db0cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:24.249277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.249259    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:24.261974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.261924    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l" pod "e38b74e4-74b9-49e3-a58c-15d004db0cb2" (UID: "e38b74e4-74b9-49e3-a58c-15d004db0cb2"). InnerVolumeSpecName "kube-api-access-dj27l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:24.293293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.293238    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca" pod "e38b74e4-74b9-49e3-a58c-15d004db0cb2" (UID: "e38b74e4-74b9-49e3-a58c-15d004db0cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:30:24.350154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.350054    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b74e4-74b9-49e3-a58c-15d004db0cb2-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:24.350154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:24.350087    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj27l\" (UniqueName: \"kubernetes.io/projected/e38b74e4-74b9-49e3-a58c-15d004db0cb2-kube-api-access-dj27l\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:25.014664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:25.014085    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"
Oct 30 17:30:25.014664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:25.014529    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4" event={"ID":"e38b74e4-74b9-49e3-a58c-15d004db0cb2","Type":"ContainerDied","Data":"9fa7e5eced60a2b970d99d9482a639c9b267e57426e79fcfb2b32f4223b2ae34"}
Oct 30 17:30:25.014664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:25.014566    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fa7e5eced60a2b970d99d9482a639c9b267e57426e79fcfb2b32f4223b2ae34"
Oct 30 17:30:25.044956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:25.044899    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:25.045147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:25.045008    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:25.525990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:25.525954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:26.044953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:26.044918    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:26.045408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:26.044976    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:27.045300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:27.045262    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:27.045796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:27.045319    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:27.337475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:27.337381    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:28.044910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:28.044873    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:28.045080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:28.044983    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:28.975674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:28.968299    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:29.045287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:29.045242    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:29.045507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:29.045313    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:29.100667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:29.096327    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:30.045054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:30.045010    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 17:30:30.045512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:30.045100    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 17:30:30.899138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:30.899104    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:31.049500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:31.049473    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:31.050012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:31.049925    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:31.050713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:31.050696    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:30:31.811894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:31.811857    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:31.825249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:31.825220    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:32.074151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:32.074061    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:30:32.194919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:32.194882    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:32.731779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:32.731742    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:34.596409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:34.596364    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:36.334587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:36.334551    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:37.999059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:37.999012    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"]
Oct 30 17:30:37.999889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:37.999864    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e38b74e4-74b9-49e3-a58c-15d004db0cb2" containerName="revoke-keys"
Oct 30 17:30:37.999889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:37.999890    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38b74e4-74b9-49e3-a58c-15d004db0cb2" containerName="revoke-keys"
Oct 30 17:30:38.000064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.000048    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38b74e4-74b9-49e3-a58c-15d004db0cb2" containerName="revoke-keys"
Oct 30 17:30:38.003960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.003937    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.045265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.045227    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"]
Oct 30 17:30:38.071027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.070990    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"]
Oct 30 17:30:38.076243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.076217    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096116    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096209    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096239    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096288    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2mj\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096354    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096445    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096468    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096521    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.096990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.096552    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.109527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.109494    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"]
Oct 30 17:30:38.129128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.129093    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:38.206920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.206817    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.207233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.207208    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.207908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.207799    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.208611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.208588    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.208751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.208692    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.208751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.208742    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.209557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.209532    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.209665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.209606    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2mj\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.209742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.209677    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.210421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210395    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.210494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210453    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdqs\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.210573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210554    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.210634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210589    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.210634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210625    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.210877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210677    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.210877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210717    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.210877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210739    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.210877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210763    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.210877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210788    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.210877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.210835    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.211964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.211517    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.212116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.211802    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.212410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.212378    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.213998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.213970    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.214456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.214432    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.232368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.232332    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.233608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.233581    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2mj\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.312313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312216    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312263    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312410    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312445    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312490    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312581    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312736    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312722    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.312838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312775    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.313049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312855    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.313049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312902    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdqs\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.313049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.312955    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.313169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.313128    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.313871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.313848    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.315127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.315100    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.315230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.315126    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.328323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.328293    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:38.336019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.335994    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.336305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.336281    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdqs\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.415213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.408419    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:38.658268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.658241    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"]
Oct 30 17:30:38.658413 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:38.658372    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1444c317_574f_48cd_a200_06c343b2d5ce.slice/crio-7dd338e10b8376dff92f345d0ae065ad2dbca1c9c1b8c0aac46dea6ae6291d70 WatchSource:0}: Error finding container 7dd338e10b8376dff92f345d0ae065ad2dbca1c9c1b8c0aac46dea6ae6291d70: Status 404 returned error can't find the container with id 7dd338e10b8376dff92f345d0ae065ad2dbca1c9c1b8c0aac46dea6ae6291d70
Oct 30 17:30:38.661162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.661131    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:38.661255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.661196    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:38.661255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.661226    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:38.750022 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:38.749985    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab480c1_9f5e_4933_968f_7b3e9feb0b75.slice/crio-7b4db08c5011be9574e9cfb79dd672fa1c4b32aed031f54fb1b71045dc3952e6 WatchSource:0}: Error finding container 7b4db08c5011be9574e9cfb79dd672fa1c4b32aed031f54fb1b71045dc3952e6: Status 404 returned error can't find the container with id 7b4db08c5011be9574e9cfb79dd672fa1c4b32aed031f54fb1b71045dc3952e6
Oct 30 17:30:38.752063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.752029    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"]
Oct 30 17:30:38.752799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.752765    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:38.752882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.752843    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:38.752882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:38.752874    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:39.129242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.129198    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" event={"ID":"cab480c1-9f5e-4933-968f-7b3e9feb0b75","Type":"ContainerStarted","Data":"331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766"}
Oct 30 17:30:39.129755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.129302    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" event={"ID":"cab480c1-9f5e-4933-968f-7b3e9feb0b75","Type":"ContainerStarted","Data":"7b4db08c5011be9574e9cfb79dd672fa1c4b32aed031f54fb1b71045dc3952e6"}
Oct 30 17:30:39.131176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.131148    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" event={"ID":"1444c317-574f-48cd-a200-06c343b2d5ce","Type":"ContainerStarted","Data":"a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e"}
Oct 30 17:30:39.131308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.131202    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" event={"ID":"1444c317-574f-48cd-a200-06c343b2d5ce","Type":"ContainerStarted","Data":"7dd338e10b8376dff92f345d0ae065ad2dbca1c9c1b8c0aac46dea6ae6291d70"}
Oct 30 17:30:39.178473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.178404    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podStartSLOduration=1.178384196 podStartE2EDuration="1.178384196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:39.175508231 +0000 UTC m=+1044.937552068" watchObservedRunningTime="2026-10-30 17:30:39.178384196 +0000 UTC m=+1044.940428010"
Oct 30 17:30:39.242594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.242195    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podStartSLOduration=2.242171632 podStartE2EDuration="2.242171632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:39.221516487 +0000 UTC m=+1044.983560302" watchObservedRunningTime="2026-10-30 17:30:39.242171632 +0000 UTC m=+1045.004215447"
Oct 30 17:30:39.331112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.331070    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:39.332610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.332584    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:39.332739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.332630    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:39.411068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.410966    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:39.412258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.412230    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:39.412386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:39.412291    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:40.051437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:40.051396    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:40.328707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:40.328593    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:40.328707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:40.328667    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:40.420969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:40.419127    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:40.420969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:40.419191    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:40.932982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:40.932949    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:41.329249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:41.329153    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:41.329249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:41.329206    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:41.416336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:41.413059    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:41.416336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:41.413136    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:42.061703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:42.054136    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:42.329731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:42.329071    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:42.329731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:42.329134    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:42.421450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:42.421338    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:42.423051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:42.421934    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:43.328738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:43.328703    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:43.329007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:43.328977    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:43.418668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:43.415130    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:43.418668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:43.415194    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:44.026666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:44.020486    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:44.151498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:44.151440    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:44.329204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:44.329116    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:44.329382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:44.329199    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:44.410812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:44.410769    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:44.411008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:44.410877    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:45.259671 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:30:45.258424    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:30:45.329131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:45.329098    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 17:30:45.329302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:45.329167    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 17:30:45.410331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:45.410297    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 17:30:45.410586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:45.410561    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 17:30:46.143710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.143628    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:46.538788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.536925    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:46.538788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.538539    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:46.546944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.546919    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 17:30:46.838783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.838039    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:46.843497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.839691    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:46.844929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:46.844884    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:30:47.584511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.584325    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"]
Oct 30 17:30:47.591029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.590999    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.606359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.605042    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"]
Oct 30 17:30:47.611257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.611211    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:30:47.611257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.611262    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:30:47.745904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.745415    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.745904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.745465    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.745904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.745600    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpnv\" (UniqueName: \"kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.847748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.847713    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.847920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.847754    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.847920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.847858    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpnv\" (UniqueName: \"kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.848845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.848827    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.851385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.851352    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.871921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.871889    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpnv\" (UniqueName: \"kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv\") pod \"maas-api-revoke-sub-9e9cb06a-lx44h\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:47.936674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:47.928935    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:48.151319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:48.151227    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:48.310349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:48.310296    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"]
Oct 30 17:30:48.311466 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:48.311422    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b0f57b_e484_4e3c_8e16_c552dcdc0c9b.slice/crio-1827b810db228596cd6ba2931a18e48c3366f47f766110a7ceefc4d74b3db333 WatchSource:0}: Error finding container 1827b810db228596cd6ba2931a18e48c3366f47f766110a7ceefc4d74b3db333: Status 404 returned error can't find the container with id 1827b810db228596cd6ba2931a18e48c3366f47f766110a7ceefc4d74b3db333
Oct 30 17:30:48.339665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:48.338705    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h" event={"ID":"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b","Type":"ContainerStarted","Data":"1827b810db228596cd6ba2931a18e48c3366f47f766110a7ceefc4d74b3db333"}
Oct 30 17:30:49.357561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:49.357519    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h" event={"ID":"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b","Type":"ContainerStarted","Data":"35f2a4b241be9736887bd2ae368ad2baf09dd3bc8bcdbd820e0e809fb21c2ee6"}
Oct 30 17:30:49.410684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:49.410156    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h" podStartSLOduration=2.410136596 podStartE2EDuration="2.410136596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:49.408933072 +0000 UTC m=+1055.170976886" watchObservedRunningTime="2026-10-30 17:30:49.410136596 +0000 UTC m=+1055.172180410"
Oct 30 17:30:50.218773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:50.218712    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:50.392160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:50.391592    5534 generic.go:356] "Generic (PLEG): container finished" podID="21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" containerID="35f2a4b241be9736887bd2ae368ad2baf09dd3bc8bcdbd820e0e809fb21c2ee6" exitCode=0
Oct 30 17:30:50.392160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:50.391784    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h" event={"ID":"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b","Type":"ContainerDied","Data":"35f2a4b241be9736887bd2ae368ad2baf09dd3bc8bcdbd820e0e809fb21c2ee6"}
Oct 30 17:30:51.639664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.639615    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:51.641349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.641327    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca\") pod \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") "
Oct 30 17:30:51.641583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.641539    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv\" (UniqueName: \"kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv\") pod \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") "
Oct 30 17:30:51.641731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.641705    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca" pod "21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" (UID: "21b0f57b-e484-4e3c-8e16-c552dcdc0c9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:51.642064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.642046    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:51.644256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.644233    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv" pod "21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" (UID: "21b0f57b-e484-4e3c-8e16-c552dcdc0c9b"). InnerVolumeSpecName "kube-api-access-hwpnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:51.743269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.743199    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca\") pod \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\" (UID: \"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b\") "
Oct 30 17:30:51.743599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.743582    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpnv\" (UniqueName: \"kubernetes.io/projected/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-kube-api-access-hwpnv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:51.745533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.745500    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca" pod "21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" (UID: "21b0f57b-e484-4e3c-8e16-c552dcdc0c9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:30:51.861797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:51.859138    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:30:52.118069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.118035    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:52.412046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.412018    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"
Oct 30 17:30:52.412046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.412038    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h" event={"ID":"21b0f57b-e484-4e3c-8e16-c552dcdc0c9b","Type":"ContainerDied","Data":"1827b810db228596cd6ba2931a18e48c3366f47f766110a7ceefc4d74b3db333"}
Oct 30 17:30:52.412296 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.412093    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1827b810db228596cd6ba2931a18e48c3366f47f766110a7ceefc4d74b3db333"
Oct 30 17:30:52.848006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.847966    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"]
Oct 30 17:30:52.849749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.849724    5534 cpu_manager.go:414] "containerMap: removing container" podUID="21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" containerName="revoke-keys"
Oct 30 17:30:52.850527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.850470    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" containerName="revoke-keys"
Oct 30 17:30:52.851062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.851041    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" containerName="revoke-keys"
Oct 30 17:30:52.860842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.860814    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.865616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.865328    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"]
Oct 30 17:30:52.876482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.876458    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"]
Oct 30 17:30:52.876716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.876563    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.897323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.897291    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"]
Oct 30 17:30:52.979970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.979925    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.979970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.979966    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980000    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980106    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980142    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980159    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkdr\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980238    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5g5\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980279    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980306    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980389    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980422    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980483    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980528    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980556    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980577    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:52.980791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980659    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980714    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:52.980791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:52.980751    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.081780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.081738    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.081988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.081852    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.081988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.081895    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082113    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082366    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082785    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082807    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082554    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082833    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.082954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082701    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082889    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.082954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082910    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkdr\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.082954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082939    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5g5\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.083160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.082983    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.083160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083001    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.083160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083044    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.083478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083456    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.083618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083567    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.083780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083603    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.083891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083617    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.083891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083719    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.083891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083840    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.084059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.083956    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.084059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.084021    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.084059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.084036    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.084229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.084060    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.084229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.084062    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.084423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.084396    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.086553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.086489    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.086860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.086818    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.086975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.086931    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.087179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.087161    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.109437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.109352    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.109573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.109503    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5g5\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.109786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.109762    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.110057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.110039    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkdr\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.185073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.185042    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:53.201220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.201167    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:53.515675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.515629    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"]
Oct 30 17:30:53.516370 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:53.516345    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1c85fd_404c_4fd6_b059_797f057acf02.slice/crio-f1efbb574c485c2ed2c4ef351f55273716c93fa4401238d0180767ae660abad7 WatchSource:0}: Error finding container f1efbb574c485c2ed2c4ef351f55273716c93fa4401238d0180767ae660abad7: Status 404 returned error can't find the container with id f1efbb574c485c2ed2c4ef351f55273716c93fa4401238d0180767ae660abad7
Oct 30 17:30:53.519571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.519540    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:53.519684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.519623    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:53.519730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.519706    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:53.544207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.544181    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"]
Oct 30 17:30:53.544366 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:30:53.544344    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf6850e_ee80_499e_98c8_346402e7b283.slice/crio-e490eb81d83bd4ddda263e1f87f208880c90673d650d1c5d66f44beb49f0faf0 WatchSource:0}: Error finding container e490eb81d83bd4ddda263e1f87f208880c90673d650d1c5d66f44beb49f0faf0: Status 404 returned error can't find the container with id e490eb81d83bd4ddda263e1f87f208880c90673d650d1c5d66f44beb49f0faf0
Oct 30 17:30:53.547526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.547489    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:53.547606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.547588    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:53.547706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.547637    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:30:53.910663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:53.909861    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:54.429980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:54.429938    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" event={"ID":"1f1c85fd-404c-4fd6-b059-797f057acf02","Type":"ContainerStarted","Data":"ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f"}
Oct 30 17:30:54.429980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:54.429987    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" event={"ID":"1f1c85fd-404c-4fd6-b059-797f057acf02","Type":"ContainerStarted","Data":"f1efbb574c485c2ed2c4ef351f55273716c93fa4401238d0180767ae660abad7"}
Oct 30 17:30:54.431335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:54.431309    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" event={"ID":"5cf6850e-ee80-499e-98c8-346402e7b283","Type":"ContainerStarted","Data":"85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c"}
Oct 30 17:30:54.431456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:54.431348    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" event={"ID":"5cf6850e-ee80-499e-98c8-346402e7b283","Type":"ContainerStarted","Data":"e490eb81d83bd4ddda263e1f87f208880c90673d650d1c5d66f44beb49f0faf0"}
Oct 30 17:30:54.480668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:54.480347    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podStartSLOduration=2.480326657 podStartE2EDuration="2.480326657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.47717855 +0000 UTC m=+1060.239222365" watchObservedRunningTime="2026-10-30 17:30:54.480326657 +0000 UTC m=+1060.242370473"
Oct 30 17:30:54.587670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:54.587269    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podStartSLOduration=2.587245256 podStartE2EDuration="2.587245256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.551143557 +0000 UTC m=+1060.313187371" watchObservedRunningTime="2026-10-30 17:30:54.587245256 +0000 UTC m=+1060.349289069"
Oct 30 17:30:55.188667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.187916    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:30:55.188667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.188092    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 17:30:55.188667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.188142    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 17:30:55.214700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.205772    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:30:55.214700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.205926    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 17:30:55.214700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.205975    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 17:30:55.872668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:55.868593    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:56.186730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:56.186095    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 17:30:56.186730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:56.186166    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 17:30:56.207671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:56.202077    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 17:30:56.207671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:56.202146    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 17:30:57.188667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:57.188080    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 17:30:57.188667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:57.188142    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 17:30:57.210676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:57.209994    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 17:30:57.210676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:57.210054    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 17:30:58.017020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:58.016987    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:30:58.186573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:58.186087    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 17:30:58.186573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:58.186164    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 17:30:58.203412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:58.202885    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 17:30:58.203412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:58.202946    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 17:30:59.189093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:59.189054    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 17:30:59.189696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:59.189663    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 17:30:59.205054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:59.205021    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 17:30:59.205295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:59.205268    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 17:30:59.952485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:30:59.952438    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:00.186106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:00.186057    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 17:31:00.186314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:00.186143    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 17:31:00.202633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:00.202530    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 17:31:00.202633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:00.202601    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 17:31:01.256084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.256059    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:31:01.257169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.257150    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:31:01.259565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.259549    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:31:01.312242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.312207    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:31:01.313013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.312992    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:31:01.314865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.314848    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:31:01.878716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:01.878675    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:03.916307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:03.916236    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:04.208754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:04.208553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:04.899978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:04.899932    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:05.013668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.012726    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:05.089668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.088878    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:31:05.112669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.099087    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:05.140666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.138754    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:05.248865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.248746    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:05.396669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.395742    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:05.910190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.910137    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:31:05.911108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.911084    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:31:05.952203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:05.952164    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.015778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.015729    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.017024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.016999    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:06.166955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.166824    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.168240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.168217    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:31:06.221677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.198788    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:31:06.221677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.199227    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:06.279768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.223160    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:31:06.279768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.223531    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:31:06.279768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.224728    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:31:06.282380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.280923    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.282380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.281529    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:31:06.442360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.419323    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.572254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.492915    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.572254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.543626    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 17:31:06.572254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.545857    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.636674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.627015    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:06.803480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.803117    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"]
Oct 30 17:31:06.828460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.828422    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:06.842677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.839062    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:31:06.920671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.844544    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:31:06.920671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.844620    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:06.920671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.873756    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"]
Oct 30 17:31:06.920671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.891588    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:06.920671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.891673    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vhs\" (UniqueName: \"kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:06.920671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.891908    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:06.953002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.950441    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.953002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.952559    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:06.953988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.953963    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:31:06.997771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.993220    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:06.997771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.993279    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vhs\" (UniqueName: \"kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:06.997771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:06.993465    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:07.033415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.031014    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:07.059038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.058958    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:07.297665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.271724    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:07.322084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.320125    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vhs\" (UniqueName: \"kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:07.470188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.470147    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:31:07.527727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.497214    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded"
Oct 30 17:31:07.527997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.497809    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:07.553405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.539244    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04764903-ztv6b\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:07.553405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.500961    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:31:07.553405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.500496    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:31:07.569988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.569937    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:07.596407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.596371    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:07.817622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.817586    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:31:07.836669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.836255    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:07.962718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.942734    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:07.962970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.962936    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"]
Oct 30 17:31:08.021571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:07.962983    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:31:08.046671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.035345    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.065769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.038492    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:08.068122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.068099    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:08.077286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.077264    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:08.078137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.078119    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:31:08.085070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.085035    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:31:08.090318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.090296    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"]
Oct 30 17:31:08.226018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.205370    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:08.250809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.238449    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.250809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.238493    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.250809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.238545    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.250809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.238605    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.250809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.244082    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.253124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.253099    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.253740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.253715    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.399445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.399411    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.399773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.399748    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.399943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.399925    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.400066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400051    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.400171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400158    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400234    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400001    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400275    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400369    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400405    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ll7\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.400526    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.401429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.401137    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.406904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.406877    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.426907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.426464    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.427349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.427305    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.429623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.429590    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.501854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.501816    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.502008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.501947    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ll7\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.503508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.503481    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.531028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.530180    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ll7\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:08.725937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.724150    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"]
Oct 30 17:31:08.735801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:08.735770    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:09.084940 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:09.084907    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d8bf3b_e41f_45d4_b3eb_8b1a459178d6.slice/crio-213285e74776883d900d8a47a136b7377a86b0987645d0f0106f6a9fa8c178fd WatchSource:0}: Error finding container 213285e74776883d900d8a47a136b7377a86b0987645d0f0106f6a9fa8c178fd: Status 404 returned error can't find the container with id 213285e74776883d900d8a47a136b7377a86b0987645d0f0106f6a9fa8c178fd
Oct 30 17:31:09.087120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.087095    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"]
Oct 30 17:31:09.088812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.088783    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:31:09.088932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.088864    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:31:09.088932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.088911    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:31:09.210252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.210214    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" event={"ID":"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6","Type":"ContainerStarted","Data":"ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48"}
Oct 30 17:31:09.210762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.210277    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" event={"ID":"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6","Type":"ContainerStarted","Data":"213285e74776883d900d8a47a136b7377a86b0987645d0f0106f6a9fa8c178fd"}
Oct 30 17:31:09.212412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.212381    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b" event={"ID":"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496","Type":"ContainerStarted","Data":"cf3d237e0e19932293b9b12ae701df66cfd6c903b65a3ce2e2fec69dc5495b29"}
Oct 30 17:31:09.212553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.212434    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b" event={"ID":"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496","Type":"ContainerStarted","Data":"61cc714ac857bcf9e5d073993402157b11b1fa161fa9ccc9bfbf9c6e208d1b51"}
Oct 30 17:31:09.260464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.260409    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podStartSLOduration=2.260386998 podStartE2EDuration="2.260386998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:09.256573223 +0000 UTC m=+1075.018617037" watchObservedRunningTime="2026-10-30 17:31:09.260386998 +0000 UTC m=+1075.022430810"
Oct 30 17:31:09.307971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.307909    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b" podStartSLOduration=3.307893145 podStartE2EDuration="3.307893145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:09.303788242 +0000 UTC m=+1075.065832081" watchObservedRunningTime="2026-10-30 17:31:09.307893145 +0000 UTC m=+1075.069936958"
Oct 30 17:31:09.738685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:09.736047    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:10.003085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.002986    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:10.111704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.109695    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"]
Oct 30 17:31:10.126431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.126402    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"]
Oct 30 17:31:10.126657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.126530    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.221175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.221140    5534 generic.go:356] "Generic (PLEG): container finished" podID="30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" containerID="cf3d237e0e19932293b9b12ae701df66cfd6c903b65a3ce2e2fec69dc5495b29" exitCode=0
Oct 30 17:31:10.221361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.221222    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b" event={"ID":"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496","Type":"ContainerDied","Data":"cf3d237e0e19932293b9b12ae701df66cfd6c903b65a3ce2e2fec69dc5495b29"}
Oct 30 17:31:10.226301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.226256    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.226492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.226471    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.226551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.226504    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6fb\" (UniqueName: \"kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.327711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.327607    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.327711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.327679    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6fb\" (UniqueName: \"kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.327934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.327834    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.328749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.328726    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.330309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.330045    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.351224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.351195    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6fb\" (UniqueName: \"kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb\") pod \"maas-api-revoke-sub-2cd98076-vgwb8\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.445433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.445393    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:10.737953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.737918    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:31:10.738140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.737976    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:31:10.795450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:10.795413    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"]
Oct 30 17:31:10.797607 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:10.797579    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b6b3d1_ce85_479f_9f10_5f7e767ff6de.slice/crio-e0e4353d4fdd0943503f44753cae1efef107332647c36e4cd602f6e512430a9f WatchSource:0}: Error finding container e0e4353d4fdd0943503f44753cae1efef107332647c36e4cd602f6e512430a9f: Status 404 returned error can't find the container with id e0e4353d4fdd0943503f44753cae1efef107332647c36e4cd602f6e512430a9f
Oct 30 17:31:11.230175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.230140    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8" event={"ID":"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de","Type":"ContainerStarted","Data":"e0e4353d4fdd0943503f44753cae1efef107332647c36e4cd602f6e512430a9f"}
Oct 30 17:31:11.443902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.443877    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:11.545958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.545921    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs\" (UniqueName: \"kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs\") pod \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") "
Oct 30 17:31:11.546128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.546031    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca\") pod \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") "
Oct 30 17:31:11.546128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.546089    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca\") pod \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\" (UID: \"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496\") "
Oct 30 17:31:11.546458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.546432    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca" pod "30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" (UID: "30f0f0bb-d14a-40eb-ae2f-d9b059c4f496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:11.546723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.546687    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:11.548155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.548132    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs" pod "30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" (UID: "30f0f0bb-d14a-40eb-ae2f-d9b059c4f496"). InnerVolumeSpecName "kube-api-access-g9vhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:11.548214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.548193    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca" pod "30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" (UID: "30f0f0bb-d14a-40eb-ae2f-d9b059c4f496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:11.647675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.647623    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vhs\" (UniqueName: \"kubernetes.io/projected/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-kube-api-access-g9vhs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:11.647675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.647671    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:11.736637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.736545    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:31:11.736637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.736607    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:31:11.855520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:11.854175    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:12.238577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.238542    5534 generic.go:356] "Generic (PLEG): container finished" podID="c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" containerID="dde9eb800cbfa28edf9d6ff28784058232657536d011b135eaa22b2a4e0e010c" exitCode=0
Oct 30 17:31:12.239071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.238731    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8" event={"ID":"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de","Type":"ContainerDied","Data":"dde9eb800cbfa28edf9d6ff28784058232657536d011b135eaa22b2a4e0e010c"}
Oct 30 17:31:12.240218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.240196    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b" event={"ID":"30f0f0bb-d14a-40eb-ae2f-d9b059c4f496","Type":"ContainerDied","Data":"61cc714ac857bcf9e5d073993402157b11b1fa161fa9ccc9bfbf9c6e208d1b51"}
Oct 30 17:31:12.240329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.240231    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61cc714ac857bcf9e5d073993402157b11b1fa161fa9ccc9bfbf9c6e208d1b51"
Oct 30 17:31:12.240329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.240276    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"
Oct 30 17:31:12.737137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.737090    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:31:12.737355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:12.737225    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:31:13.541159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.541131    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:13.707372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.707325    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca\") pod \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") "
Oct 30 17:31:13.707593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.707423    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb\" (UniqueName: \"kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb\") pod \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") "
Oct 30 17:31:13.707593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.707538    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca\") pod \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\" (UID: \"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de\") "
Oct 30 17:31:13.708205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.708180    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca" pod "c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" (UID: "c0b6b3d1-ce85-479f-9f10-5f7e767ff6de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:13.710670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.710478    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca" pod "c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" (UID: "c0b6b3d1-ce85-479f-9f10-5f7e767ff6de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:13.711122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.711099    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb" pod "c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" (UID: "c0b6b3d1-ce85-479f-9f10-5f7e767ff6de"). InnerVolumeSpecName "kube-api-access-4v6fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:13.736420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.736385    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:31:13.736594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.736448    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:31:13.809921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.809867    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:13.809921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.809918    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:13.810134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.809976    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6fb\" (UniqueName: \"kubernetes.io/projected/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de-kube-api-access-4v6fb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:13.994719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:13.992027    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:14.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.310225    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"
Oct 30 17:31:14.310496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.310230    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8" event={"ID":"c0b6b3d1-ce85-479f-9f10-5f7e767ff6de","Type":"ContainerDied","Data":"e0e4353d4fdd0943503f44753cae1efef107332647c36e4cd602f6e512430a9f"}
Oct 30 17:31:14.310594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.310552    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e4353d4fdd0943503f44753cae1efef107332647c36e4cd602f6e512430a9f"
Oct 30 17:31:14.448896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.448861    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"]
Oct 30 17:31:14.449581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.449564    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" containerName="revoke-keys"
Oct 30 17:31:14.449637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.449583    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" containerName="revoke-keys"
Oct 30 17:31:14.449637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.449617    5534 cpu_manager.go:414] "containerMap: removing container" podUID="30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" containerName="revoke-keys"
Oct 30 17:31:14.449637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.449622    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" containerName="revoke-keys"
Oct 30 17:31:14.449781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.449746    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" containerName="revoke-keys"
Oct 30 17:31:14.449781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.449759    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" containerName="revoke-keys"
Oct 30 17:31:14.476667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.471963    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.494228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.491969    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:14.494228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.492687    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:31:14.498500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.498471    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"]
Oct 30 17:31:14.622668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.622625    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.623137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.622711    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.623137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.622742    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44btn\" (UniqueName: \"kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.724472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.724432    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.724630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.724497    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.724630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.724527    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44btn\" (UniqueName: \"kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.725420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.725386    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.727041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.727019    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.736226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.736198    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:31:14.736319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.736271    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:31:14.748142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.748113    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44btn\" (UniqueName: \"kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn\") pod \"maas-api-revoke-sub-651e6b40-hhrqz\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:14.817054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:14.817022    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:15.173965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:15.173495    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"]
Oct 30 17:31:15.176320 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:15.176293    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e3b990_44d6_41a0_ad6a_03531d2a6327.slice/crio-4dace1e6ebc3c6b3f6cffaa4301ab43f862e8fc9a83e1c807a54b6f0dfb5b264 WatchSource:0}: Error finding container 4dace1e6ebc3c6b3f6cffaa4301ab43f862e8fc9a83e1c807a54b6f0dfb5b264: Status 404 returned error can't find the container with id 4dace1e6ebc3c6b3f6cffaa4301ab43f862e8fc9a83e1c807a54b6f0dfb5b264
Oct 30 17:31:15.245136 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:31:15.245093    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:31:15.320496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:15.320457    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz" event={"ID":"d3e3b990-44d6-41a0-ad6a-03531d2a6327","Type":"ContainerStarted","Data":"4dace1e6ebc3c6b3f6cffaa4301ab43f862e8fc9a83e1c807a54b6f0dfb5b264"}
Oct 30 17:31:15.742360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:15.742260    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:31:15.743226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:15.742981    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:31:15.852672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:15.845738    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:16.363665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:16.360597    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz" event={"ID":"d3e3b990-44d6-41a0-ad6a-03531d2a6327","Type":"ContainerStarted","Data":"a7e3969ac28da47fecdf4d41e0933c1289f4f768a2467f224dca1b0388a04bfb"}
Oct 30 17:31:17.402671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.397595    5534 generic.go:356] "Generic (PLEG): container finished" podID="d3e3b990-44d6-41a0-ad6a-03531d2a6327" containerID="a7e3969ac28da47fecdf4d41e0933c1289f4f768a2467f224dca1b0388a04bfb" exitCode=0
Oct 30 17:31:17.402671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.397812    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz" event={"ID":"d3e3b990-44d6-41a0-ad6a-03531d2a6327","Type":"ContainerDied","Data":"a7e3969ac28da47fecdf4d41e0933c1289f4f768a2467f224dca1b0388a04bfb"}
Oct 30 17:31:17.864389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.863723    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:31:17.864389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.863911    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:31:17.873482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.873372    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:31:17.899908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.899862    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:17.911672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.909739    5534 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:31:17.911672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.909804    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:17.964805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:17.964765    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:31:18.040329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.040288    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.084771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.084724    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.086263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.086234    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:31:18.132855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.132745    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.155997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.155957    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:31:18.234077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.234040    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:18.260669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.254923    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:18.337247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.313959    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.337247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.316614    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:31:18.529005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.528905    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.591931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.591778    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.918673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.899980    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:18.955669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.954202    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:18.955669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:18.954823    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:19.042870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.042829    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:19.057672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.047587    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:19.129156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.129116    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:19.130151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.130130    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:19.174678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.174540    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:19.175562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.175540    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:19.183310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.183121    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:19.199664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.198299    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:19.220985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.220950    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:19.226905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.226476    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:19.248770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.248744    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:19.377086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.377052    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn\" (UniqueName: \"kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn\") pod \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") "
Oct 30 17:31:19.377254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.377103    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca\") pod \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") "
Oct 30 17:31:19.377254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.377228    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca\") pod \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\" (UID: \"d3e3b990-44d6-41a0-ad6a-03531d2a6327\") "
Oct 30 17:31:19.377686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.377655    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca" pod "d3e3b990-44d6-41a0-ad6a-03531d2a6327" (UID: "d3e3b990-44d6-41a0-ad6a-03531d2a6327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:19.377970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.377946    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:19.380139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.380110    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn" pod "d3e3b990-44d6-41a0-ad6a-03531d2a6327" (UID: "d3e3b990-44d6-41a0-ad6a-03531d2a6327"). InnerVolumeSpecName "kube-api-access-44btn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:19.380139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.380124    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca" pod "d3e3b990-44d6-41a0-ad6a-03531d2a6327" (UID: "d3e3b990-44d6-41a0-ad6a-03531d2a6327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:19.479513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.479432    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44btn\" (UniqueName: \"kubernetes.io/projected/d3e3b990-44d6-41a0-ad6a-03531d2a6327-kube-api-access-44btn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:19.479701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:19.479681    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e3b990-44d6-41a0-ad6a-03531d2a6327-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:20.182796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:20.182769    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"
Oct 30 17:31:20.182796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:20.182783    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz" event={"ID":"d3e3b990-44d6-41a0-ad6a-03531d2a6327","Type":"ContainerDied","Data":"4dace1e6ebc3c6b3f6cffaa4301ab43f862e8fc9a83e1c807a54b6f0dfb5b264"}
Oct 30 17:31:20.183347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:20.182845    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dace1e6ebc3c6b3f6cffaa4301ab43f862e8fc9a83e1c807a54b6f0dfb5b264"
Oct 30 17:31:20.502048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:20.501935    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:22.125894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.125853    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"]
Oct 30 17:31:22.126727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.126704    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d3e3b990-44d6-41a0-ad6a-03531d2a6327" containerName="revoke-keys"
Oct 30 17:31:22.126845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.126727    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e3b990-44d6-41a0-ad6a-03531d2a6327" containerName="revoke-keys"
Oct 30 17:31:22.126937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.126923    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e3b990-44d6-41a0-ad6a-03531d2a6327" containerName="revoke-keys"
Oct 30 17:31:22.131079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.131059    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.149392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.149362    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"]
Oct 30 17:31:22.157325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.157303    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:22.157471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.157340    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:31:22.219030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.218994    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.219203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.219131    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.219358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.219339    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmz9\" (UniqueName: \"kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.296999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.296947    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:22.320389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.320353    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmz9\" (UniqueName: \"kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.320581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.320462    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.320581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.320551    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.321232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.321203    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.323107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.323086    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.352596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.352549    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmz9\" (UniqueName: \"kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9\") pod \"maas-api-revoke-sub-9824352d-tk6pc\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.444375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.444293    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:22.757212 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:22.757182    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode189b59e_f5a3_4f79_bfe3_2ffa30c264f9.slice/crio-7f9b8955fe612f0b06c4b6963873d225c10bbc2edcc23a553580e3a6aad41d1e WatchSource:0}: Error finding container 7f9b8955fe612f0b06c4b6963873d225c10bbc2edcc23a553580e3a6aad41d1e: Status 404 returned error can't find the container with id 7f9b8955fe612f0b06c4b6963873d225c10bbc2edcc23a553580e3a6aad41d1e
Oct 30 17:31:22.759320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:22.759293    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"]
Oct 30 17:31:23.206236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.206194    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc" event={"ID":"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9","Type":"ContainerStarted","Data":"4c94695aa753a1358700c33bc6545c8d6e6c94cb467e7c651251f9a1a755cb7b"}
Oct 30 17:31:23.206746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.206260    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc" event={"ID":"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9","Type":"ContainerStarted","Data":"7f9b8955fe612f0b06c4b6963873d225c10bbc2edcc23a553580e3a6aad41d1e"}
Oct 30 17:31:23.213783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.213753    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"]
Oct 30 17:31:23.218732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.218714    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241231    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241289    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241353    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241530    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241588    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241786    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241806    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241867    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqzw\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.241927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.241884    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.250267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.250236    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"]
Oct 30 17:31:23.310144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.310086    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc" podStartSLOduration=1.310068682 podStartE2EDuration="1.310068682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:23.263540462 +0000 UTC m=+1089.025584278" watchObservedRunningTime="2026-10-30 17:31:23.310068682 +0000 UTC m=+1089.072112495"
Oct 30 17:31:23.343929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.343900    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344043    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344083    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344102    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344179    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344242    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344298    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344323    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344356    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344383    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqzw\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344573    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344619    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.344779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.344760    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.345228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.345198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.346931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.346905    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.347116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.347098    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.367707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.367678    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.367861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.367757    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqzw\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.536560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.536527    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:23.866435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.866405    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"]
Oct 30 17:31:23.867213 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:23.867182    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3885c7_5694_4c68_9deb_8cc6e801d105.slice/crio-ec0deaa939d408d2f079ecc663124d87bad69b5df7d8c148cbd0a3a90c46fb59 WatchSource:0}: Error finding container ec0deaa939d408d2f079ecc663124d87bad69b5df7d8c148cbd0a3a90c46fb59: Status 404 returned error can't find the container with id ec0deaa939d408d2f079ecc663124d87bad69b5df7d8c148cbd0a3a90c46fb59
Oct 30 17:31:23.870196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.870155    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:31:23.870303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.870252    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:31:23.870303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:23.870294    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:31:24.079788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.079754    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:24.215188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.215076    5534 generic.go:356] "Generic (PLEG): container finished" podID="e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" containerID="4c94695aa753a1358700c33bc6545c8d6e6c94cb467e7c651251f9a1a755cb7b" exitCode=0
Oct 30 17:31:24.215188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.215173    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc" event={"ID":"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9","Type":"ContainerDied","Data":"4c94695aa753a1358700c33bc6545c8d6e6c94cb467e7c651251f9a1a755cb7b"}
Oct 30 17:31:24.217298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.217274    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" event={"ID":"8f3885c7-5694-4c68-9deb-8cc6e801d105","Type":"ContainerStarted","Data":"95652db4d6155482d724f056c3288b1573b013911a130a6c0048cf2f1b999a74"}
Oct 30 17:31:24.217437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.217315    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" event={"ID":"8f3885c7-5694-4c68-9deb-8cc6e801d105","Type":"ContainerStarted","Data":"ec0deaa939d408d2f079ecc663124d87bad69b5df7d8c148cbd0a3a90c46fb59"}
Oct 30 17:31:24.309124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.309067    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podStartSLOduration=1.309047622 podStartE2EDuration="1.309047622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:24.304108744 +0000 UTC m=+1090.066152568" watchObservedRunningTime="2026-10-30 17:31:24.309047622 +0000 UTC m=+1090.071091437"
Oct 30 17:31:24.536806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.536685    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:24.538469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.538437    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:24.538625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:24.538509    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:25.441587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.441537    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:25.510719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.510104    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca\") pod \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") "
Oct 30 17:31:25.510719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.510355    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca\") pod \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") "
Oct 30 17:31:25.510719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.510392    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9\" (UniqueName: \"kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9\") pod \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\" (UID: \"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9\") "
Oct 30 17:31:25.513681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.512411    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca" pod "e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" (UID: "e189b59e-f5a3-4f79-bfe3-2ffa30c264f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:25.517557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.517180    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca" pod "e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" (UID: "e189b59e-f5a3-4f79-bfe3-2ffa30c264f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:25.518828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.518796    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9" pod "e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" (UID: "e189b59e-f5a3-4f79-bfe3-2ffa30c264f9"). InnerVolumeSpecName "kube-api-access-7cmz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:25.538076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.538035    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:25.538233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.538164    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:25.612277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.612242    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:25.612277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.612280    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:25.612277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.612292    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cmz9\" (UniqueName: \"kubernetes.io/projected/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9-kube-api-access-7cmz9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:25.937706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:25.937672    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:26.237314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:26.237239    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"
Oct 30 17:31:26.239748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:26.239710    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc" event={"ID":"e189b59e-f5a3-4f79-bfe3-2ffa30c264f9","Type":"ContainerDied","Data":"7f9b8955fe612f0b06c4b6963873d225c10bbc2edcc23a553580e3a6aad41d1e"}
Oct 30 17:31:26.239862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:26.239767    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9b8955fe612f0b06c4b6963873d225c10bbc2edcc23a553580e3a6aad41d1e"
Oct 30 17:31:26.537455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:26.537377    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:26.537455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:26.537430    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:27.537478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.537444    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:27.537884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.537497    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:27.629761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.629710    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"]
Oct 30 17:31:27.630492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.630468    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" containerName="revoke-keys"
Oct 30 17:31:27.630492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.630493    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" containerName="revoke-keys"
Oct 30 17:31:27.630719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.630702    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" containerName="revoke-keys"
Oct 30 17:31:27.634935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.634913    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.664944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.664911    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:27.667066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.666830    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"]
Oct 30 17:31:27.667291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.667003    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:31:27.743537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.743502    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.743746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.743729    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qt9\" (UniqueName: \"kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.743866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.743848    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.816821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.816710    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:27.844862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.844819    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.845040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.844989    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qt9\" (UniqueName: \"kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.845109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.845093    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.845860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.845832    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.847355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.847331    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.893692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.888882    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qt9\" (UniqueName: \"kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9\") pod \"maas-api-revoke-sub-a44d8871-69dr4\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:27.950502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:27.950472    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:28.303242 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:28.303208    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea60540_ed8a_4686_bedf_fb7408634166.slice/crio-087efb06587b76a3e9fd947bd659312500eef9578619b1c17c850ecab1a26f24 WatchSource:0}: Error finding container 087efb06587b76a3e9fd947bd659312500eef9578619b1c17c850ecab1a26f24: Status 404 returned error can't find the container with id 087efb06587b76a3e9fd947bd659312500eef9578619b1c17c850ecab1a26f24
Oct 30 17:31:28.303780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:28.303632    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"]
Oct 30 17:31:28.537167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:28.537136    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:28.537309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:28.537200    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:29.080730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.080296    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:29.311923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.311880    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4" event={"ID":"bea60540-ed8a-4686-bedf-fb7408634166","Type":"ContainerStarted","Data":"4bdc6ebc4c9d52d3b38bbc2136449f5a1744543ced87ee33160c70b1bf5aab65"}
Oct 30 17:31:29.312071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.311953    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4" event={"ID":"bea60540-ed8a-4686-bedf-fb7408634166","Type":"ContainerStarted","Data":"087efb06587b76a3e9fd947bd659312500eef9578619b1c17c850ecab1a26f24"}
Oct 30 17:31:29.393774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.393691    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4" podStartSLOduration=2.393667631 podStartE2EDuration="2.393667631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:29.35219508 +0000 UTC m=+1095.114238892" watchObservedRunningTime="2026-10-30 17:31:29.393667631 +0000 UTC m=+1095.155711444"
Oct 30 17:31:29.537286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.537251    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:29.537462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.537305    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:29.630767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:29.630724    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:30.322381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:30.322346    5534 generic.go:356] "Generic (PLEG): container finished" podID="bea60540-ed8a-4686-bedf-fb7408634166" containerID="4bdc6ebc4c9d52d3b38bbc2136449f5a1744543ced87ee33160c70b1bf5aab65" exitCode=0
Oct 30 17:31:30.322872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:30.322436    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4" event={"ID":"bea60540-ed8a-4686-bedf-fb7408634166","Type":"ContainerDied","Data":"4bdc6ebc4c9d52d3b38bbc2136449f5a1744543ced87ee33160c70b1bf5aab65"}
Oct 30 17:31:30.536958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:30.536926    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 17:31:30.537125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:30.536993    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 17:31:31.428481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.428419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:31.530395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.530366    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:31.544118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.542772    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:31.544444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.544360    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:31.549709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.549670    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:31:31.706325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.695168    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca\") pod \"bea60540-ed8a-4686-bedf-fb7408634166\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") "
Oct 30 17:31:31.773731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.769258    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca\") pod \"bea60540-ed8a-4686-bedf-fb7408634166\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") "
Oct 30 17:31:31.773731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.769405    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9\" (UniqueName: \"kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9\") pod \"bea60540-ed8a-4686-bedf-fb7408634166\" (UID: \"bea60540-ed8a-4686-bedf-fb7408634166\") "
Oct 30 17:31:31.785240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.785205    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca" pod "bea60540-ed8a-4686-bedf-fb7408634166" (UID: "bea60540-ed8a-4686-bedf-fb7408634166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:31.851299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.851013    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca" pod "bea60540-ed8a-4686-bedf-fb7408634166" (UID: "bea60540-ed8a-4686-bedf-fb7408634166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:31.873079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.873045    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:31.873293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.873274    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea60540-ed8a-4686-bedf-fb7408634166-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:31.876817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.875826    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9" pod "bea60540-ed8a-4686-bedf-fb7408634166" (UID: "bea60540-ed8a-4686-bedf-fb7408634166"). InnerVolumeSpecName "kube-api-access-l5qt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:31.975727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:31.975621    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qt9\" (UniqueName: \"kubernetes.io/projected/bea60540-ed8a-4686-bedf-fb7408634166-kube-api-access-l5qt9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:32.361399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:32.360105    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"
Oct 30 17:31:32.361399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:32.360938    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4" event={"ID":"bea60540-ed8a-4686-bedf-fb7408634166","Type":"ContainerDied","Data":"087efb06587b76a3e9fd947bd659312500eef9578619b1c17c850ecab1a26f24"}
Oct 30 17:31:32.361399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:32.360997    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087efb06587b76a3e9fd947bd659312500eef9578619b1c17c850ecab1a26f24"
Oct 30 17:31:33.501787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:33.501736    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:31:33.520785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:33.520731    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:31:33.840854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:33.840817    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:31:33.841432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:33.841090    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:34.083010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.081758    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:34.118850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.118744    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:31:34.226679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.225728    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:34.242670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.239740    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:34.251548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.251509    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:31:34.522768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.522660    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:34.530667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.523371    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:34.667670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.652162    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:31:34.983183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.983150    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:31:35.092671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.807832    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.092671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:34.923273    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:35.126834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.126781    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.183677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.162627    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:31:35.252732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.208883    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:31:35.329586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.302193    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:31:35.329586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.228059    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:31:35.329586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.260458    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.330964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.277291    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.357670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.332828    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:35.357670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.332927    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:35.357670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.289407    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.357670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.228136    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.556756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.555785    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.584673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.580909    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:35.625681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.622842    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:35.937679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:35.936742    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.190723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.187470    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.317669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.317567    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.342672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.319338    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:31:36.342672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.320578    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.342672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.326115    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:36.342672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.338567    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:31:36.526963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.523276    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.560425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.560381    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.591532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.591495    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.592032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.592007    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:31:36.592353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.592333    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 30 17:31:36.592547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.592514    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:36.758226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.758199    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:31:36.762793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.762767    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:31:36.767489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.766732    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:31:36.784241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.783737    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:36.856032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.855999    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:31:36.913046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.913017    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:31:36.915991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.915966    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:36.926299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.926274    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:31:36.947303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.947273    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:36.973157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:36.971086    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:37.523414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:37.523377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:38.035534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.035478    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"]
Oct 30 17:31:38.036630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.036601    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bea60540-ed8a-4686-bedf-fb7408634166" containerName="revoke-keys"
Oct 30 17:31:38.036630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.036626    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea60540-ed8a-4686-bedf-fb7408634166" containerName="revoke-keys"
Oct 30 17:31:38.036859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.036850    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea60540-ed8a-4686-bedf-fb7408634166" containerName="revoke-keys"
Oct 30 17:31:38.041080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.041056    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"]
Oct 30 17:31:38.041206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.041191    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.045456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.045431    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.060741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.060703    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"]
Oct 30 17:31:38.063903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.060981    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:38.064015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.061137    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:31:38.071941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.071909    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"]
Oct 30 17:31:38.113175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.113135    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.113356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.113290    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.113410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.113369    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.113539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.113519    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5x8\" (UniqueName: \"kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.113694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.113663    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.113825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.113806    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8476\" (UniqueName: \"kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.214964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.214927    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5x8\" (UniqueName: \"kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.215168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215018    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.215168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215062    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8476\" (UniqueName: \"kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.215168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215105    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.215168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215163    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.215377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215192    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.215925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215847    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.215925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.215847    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.217625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.217601    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.217625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.217623    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.238388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.238361    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8476\" (UniqueName: \"kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476\") pod \"maas-api-revoke-sub-2526c403-gpwmq\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.238495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.238364    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5x8\" (UniqueName: \"kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8\") pod \"maas-api-revoke-sub-f658dee5-l2gl9\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.358309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.358271    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:38.371850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.371824    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:38.679812 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:38.679778    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f8600d_e813_41ab_8829_f01b51a3a521.slice/crio-e9390c0907a1e9975f73591a3d213d2a9938ae6ee8cdd311edcb9f408c4ecd03 WatchSource:0}: Error finding container e9390c0907a1e9975f73591a3d213d2a9938ae6ee8cdd311edcb9f408c4ecd03: Status 404 returned error can't find the container with id e9390c0907a1e9975f73591a3d213d2a9938ae6ee8cdd311edcb9f408c4ecd03
Oct 30 17:31:38.680085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.680064    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"]
Oct 30 17:31:38.694015 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:38.693983    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec388735_c715_441b_8540_15f79b44f44e.slice/crio-74ef6461a1bb0921cd095cfdb5c20b065e3408e219a86c0ad6d7eb4654b4ea60 WatchSource:0}: Error finding container 74ef6461a1bb0921cd095cfdb5c20b065e3408e219a86c0ad6d7eb4654b4ea60: Status 404 returned error can't find the container with id 74ef6461a1bb0921cd095cfdb5c20b065e3408e219a86c0ad6d7eb4654b4ea60
Oct 30 17:31:38.697544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:38.697514    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"]
Oct 30 17:31:39.016887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:39.016850    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq" event={"ID":"c6f8600d-e813-41ab-8829-f01b51a3a521","Type":"ContainerStarted","Data":"e9390c0907a1e9975f73591a3d213d2a9938ae6ee8cdd311edcb9f408c4ecd03"}
Oct 30 17:31:39.018677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:39.018632    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9" event={"ID":"ec388735-c715-441b-8540-15f79b44f44e","Type":"ContainerStarted","Data":"ffe9b7c9f18e0e22cb85b451f4dd63ada00906ab8e87defa722ee8e834c2fcfc"}
Oct 30 17:31:39.018814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:39.018691    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9" event={"ID":"ec388735-c715-441b-8540-15f79b44f44e","Type":"ContainerStarted","Data":"74ef6461a1bb0921cd095cfdb5c20b065e3408e219a86c0ad6d7eb4654b4ea60"}
Oct 30 17:31:39.061599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:39.061546    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9" podStartSLOduration=1.061532371 podStartE2EDuration="1.061532371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:39.061107645 +0000 UTC m=+1104.823151460" watchObservedRunningTime="2026-10-30 17:31:39.061532371 +0000 UTC m=+1104.823576243"
Oct 30 17:31:39.213272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:39.213230    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:40.027416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:40.027378    5534 generic.go:356] "Generic (PLEG): container finished" podID="ec388735-c715-441b-8540-15f79b44f44e" containerID="ffe9b7c9f18e0e22cb85b451f4dd63ada00906ab8e87defa722ee8e834c2fcfc" exitCode=0
Oct 30 17:31:40.027638 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:40.027472    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9" event={"ID":"ec388735-c715-441b-8540-15f79b44f44e","Type":"ContainerDied","Data":"ffe9b7c9f18e0e22cb85b451f4dd63ada00906ab8e87defa722ee8e834c2fcfc"}
Oct 30 17:31:40.029353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:40.029324    5534 generic.go:356] "Generic (PLEG): container finished" podID="c6f8600d-e813-41ab-8829-f01b51a3a521" containerID="599ce212844532f60f7690e33244f7f1922c6b606f885a974ed3b8d1c50ed02c" exitCode=0
Oct 30 17:31:40.029458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:40.029397    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq" event={"ID":"c6f8600d-e813-41ab-8829-f01b51a3a521","Type":"ContainerDied","Data":"599ce212844532f60f7690e33244f7f1922c6b606f885a974ed3b8d1c50ed02c"}
Oct 30 17:31:40.910224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:40.910186    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:41.241597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.241570    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:41.267174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.267149    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:41.364051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364015    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca\") pod \"c6f8600d-e813-41ab-8829-f01b51a3a521\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") "
Oct 30 17:31:41.364219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364167    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476\" (UniqueName: \"kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476\") pod \"c6f8600d-e813-41ab-8829-f01b51a3a521\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") "
Oct 30 17:31:41.364302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364227    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca\") pod \"ec388735-c715-441b-8540-15f79b44f44e\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") "
Oct 30 17:31:41.364391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364368    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca\") pod \"ec388735-c715-441b-8540-15f79b44f44e\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") "
Oct 30 17:31:41.364541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364433    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8\" (UniqueName: \"kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8\") pod \"ec388735-c715-441b-8540-15f79b44f44e\" (UID: \"ec388735-c715-441b-8540-15f79b44f44e\") "
Oct 30 17:31:41.364608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364551    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca\") pod \"c6f8600d-e813-41ab-8829-f01b51a3a521\" (UID: \"c6f8600d-e813-41ab-8829-f01b51a3a521\") "
Oct 30 17:31:41.364723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.364693    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca" pod "ec388735-c715-441b-8540-15f79b44f44e" (UID: "ec388735-c715-441b-8540-15f79b44f44e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:41.365187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.365158    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca" pod "c6f8600d-e813-41ab-8829-f01b51a3a521" (UID: "c6f8600d-e813-41ab-8829-f01b51a3a521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:41.365288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.365232    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec388735-c715-441b-8540-15f79b44f44e-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:41.367254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.367231    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8" pod "ec388735-c715-441b-8540-15f79b44f44e" (UID: "ec388735-c715-441b-8540-15f79b44f44e"). InnerVolumeSpecName "kube-api-access-fz5x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:41.367359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.367255    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca" pod "c6f8600d-e813-41ab-8829-f01b51a3a521" (UID: "c6f8600d-e813-41ab-8829-f01b51a3a521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:41.367359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.367263    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476" pod "c6f8600d-e813-41ab-8829-f01b51a3a521" (UID: "c6f8600d-e813-41ab-8829-f01b51a3a521"). InnerVolumeSpecName "kube-api-access-x8476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:41.367486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.367467    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca" pod "ec388735-c715-441b-8540-15f79b44f44e" (UID: "ec388735-c715-441b-8540-15f79b44f44e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:41.466897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.466819    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8476\" (UniqueName: \"kubernetes.io/projected/c6f8600d-e813-41ab-8829-f01b51a3a521-kube-api-access-x8476\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:41.466897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.466847    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec388735-c715-441b-8540-15f79b44f44e-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:41.466897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.466856    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5x8\" (UniqueName: \"kubernetes.io/projected/ec388735-c715-441b-8540-15f79b44f44e-kube-api-access-fz5x8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:41.466897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.466866    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:41.466897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:41.466876    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8600d-e813-41ab-8829-f01b51a3a521-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:42.046834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.046804    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"
Oct 30 17:31:42.047265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.046827    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9" event={"ID":"ec388735-c715-441b-8540-15f79b44f44e","Type":"ContainerDied","Data":"74ef6461a1bb0921cd095cfdb5c20b065e3408e219a86c0ad6d7eb4654b4ea60"}
Oct 30 17:31:42.047265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.046882    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74ef6461a1bb0921cd095cfdb5c20b065e3408e219a86c0ad6d7eb4654b4ea60"
Oct 30 17:31:42.048439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.048414    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq" event={"ID":"c6f8600d-e813-41ab-8829-f01b51a3a521","Type":"ContainerDied","Data":"e9390c0907a1e9975f73591a3d213d2a9938ae6ee8cdd311edcb9f408c4ecd03"}
Oct 30 17:31:42.048545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.048445    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"
Oct 30 17:31:42.048545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.048465    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9390c0907a1e9975f73591a3d213d2a9938ae6ee8cdd311edcb9f408c4ecd03"
Oct 30 17:31:42.720040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:42.719996    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:44.217513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:44.217468    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:44.371741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:44.371702    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:45.249086 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:31:45.249043    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:31:45.976629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:45.976590    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:47.715349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:47.715308    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:48.885454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.885415    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"]
Oct 30 17:31:48.886429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.886409    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c6f8600d-e813-41ab-8829-f01b51a3a521" containerName="revoke-keys"
Oct 30 17:31:48.886518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.886431    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f8600d-e813-41ab-8829-f01b51a3a521" containerName="revoke-keys"
Oct 30 17:31:48.886518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.886453    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ec388735-c715-441b-8540-15f79b44f44e" containerName="revoke-keys"
Oct 30 17:31:48.886518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.886461    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec388735-c715-441b-8540-15f79b44f44e" containerName="revoke-keys"
Oct 30 17:31:48.886700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.886609    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f8600d-e813-41ab-8829-f01b51a3a521" containerName="revoke-keys"
Oct 30 17:31:48.886700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.886629    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec388735-c715-441b-8540-15f79b44f44e" containerName="revoke-keys"
Oct 30 17:31:48.892273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.892250    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:48.912165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.912053    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"]
Oct 30 17:31:48.917332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.917297    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:31:48.917910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.917887    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:48.986158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.986119    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldcg\" (UniqueName: \"kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:48.986158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.986175    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:48.986468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:48.986439    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.088251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.088213    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldcg\" (UniqueName: \"kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.088410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.088272    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.088482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.088462    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.089138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.089105    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.091048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.091024    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.118601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.118573    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldcg\" (UniqueName: \"kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg\") pod \"maas-api-revoke-sub-5736bb01-9qpg7\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.210943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.210872    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:49.423062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.423028    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:49.532382 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:49.531449    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb2dc2f_26db_4b86_b69d_1f7b6ca12b1a.slice/crio-c372f3ab7e9cfe0e8fc5a20c2e243161ce52906fcf36d723809dbbcfb9181627 WatchSource:0}: Error finding container c372f3ab7e9cfe0e8fc5a20c2e243161ce52906fcf36d723809dbbcfb9181627: Status 404 returned error can't find the container with id c372f3ab7e9cfe0e8fc5a20c2e243161ce52906fcf36d723809dbbcfb9181627
Oct 30 17:31:49.533288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:49.533265    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"]
Oct 30 17:31:50.122954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.122920    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7" event={"ID":"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a","Type":"ContainerStarted","Data":"e80c2ac1b6b188b35f324f054fb6332e9e9c7846d5e61c0c33217b7d9744e694"}
Oct 30 17:31:50.123363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.122981    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7" event={"ID":"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a","Type":"ContainerStarted","Data":"c372f3ab7e9cfe0e8fc5a20c2e243161ce52906fcf36d723809dbbcfb9181627"}
Oct 30 17:31:50.167065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.167010    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7" podStartSLOduration=2.166995515 podStartE2EDuration="2.166995515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:50.162933458 +0000 UTC m=+1115.924977272" watchObservedRunningTime="2026-10-30 17:31:50.166995515 +0000 UTC m=+1115.929039344"
Oct 30 17:31:50.394701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.394663    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"]
Oct 30 17:31:50.398997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.398971    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.419346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.419314    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"]
Oct 30 17:31:50.508650 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.508616    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.508882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.508855    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwgd\" (UniqueName: \"kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.508992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.508975    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.610125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.610092    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwgd\" (UniqueName: \"kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.610294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.610190    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.610406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.610387    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.611007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.610976    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.612838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.612818    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.633589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.633560    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwgd\" (UniqueName: \"kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd\") pod \"maas-api-revoke-sub-e6d8c5cc-sdd9r\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:50.721617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:50.721509    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:51.042370 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:51.042332    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd0e7ac_603f_4ef3_8209_93563d372317.slice/crio-b77fb076fc9b0438a295ad2fd81ad1ab7a29a711b4bf5006a8250ee8835011a0 WatchSource:0}: Error finding container b77fb076fc9b0438a295ad2fd81ad1ab7a29a711b4bf5006a8250ee8835011a0: Status 404 returned error can't find the container with id b77fb076fc9b0438a295ad2fd81ad1ab7a29a711b4bf5006a8250ee8835011a0
Oct 30 17:31:51.043436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:51.043415    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"]
Oct 30 17:31:51.132160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:51.132127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r" event={"ID":"8dd0e7ac-603f-4ef3-8209-93563d372317","Type":"ContainerStarted","Data":"b77fb076fc9b0438a295ad2fd81ad1ab7a29a711b4bf5006a8250ee8835011a0"}
Oct 30 17:31:51.133815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:51.133787    5534 generic.go:356] "Generic (PLEG): container finished" podID="0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" containerID="e80c2ac1b6b188b35f324f054fb6332e9e9c7846d5e61c0c33217b7d9744e694" exitCode=0
Oct 30 17:31:51.133963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:51.133873    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7" event={"ID":"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a","Type":"ContainerDied","Data":"e80c2ac1b6b188b35f324f054fb6332e9e9c7846d5e61c0c33217b7d9744e694"}
Oct 30 17:31:51.174988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:51.174951    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:52.143424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:52.143385    5534 generic.go:356] "Generic (PLEG): container finished" podID="8dd0e7ac-603f-4ef3-8209-93563d372317" containerID="67212b1427563c5d1746cda395d2317ce7b39b9af84921e0c7c7618c25e475dd" exitCode=0
Oct 30 17:31:52.143913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:52.143479    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r" event={"ID":"8dd0e7ac-603f-4ef3-8209-93563d372317","Type":"ContainerDied","Data":"67212b1427563c5d1746cda395d2317ce7b39b9af84921e0c7c7618c25e475dd"}
Oct 30 17:31:52.586074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:52.586035    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:53.145076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.145036    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg\" (UniqueName: \"kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg\") pod \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") "
Oct 30 17:31:53.166906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.166871    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca\") pod \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") "
Oct 30 17:31:53.167329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.167300    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca\") pod \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\" (UID: \"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a\") "
Oct 30 17:31:53.168984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.168957    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca" pod "0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" (UID: "0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:53.181845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.181820    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"
Oct 30 17:31:53.198791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.198753    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7" event={"ID":"0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a","Type":"ContainerDied","Data":"c372f3ab7e9cfe0e8fc5a20c2e243161ce52906fcf36d723809dbbcfb9181627"}
Oct 30 17:31:53.199036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.199016    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c372f3ab7e9cfe0e8fc5a20c2e243161ce52906fcf36d723809dbbcfb9181627"
Oct 30 17:31:53.240945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.240907    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca" pod "0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" (UID: "0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:53.275882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.274463    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:53.275882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.274514    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.275882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.274539    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.275882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.274757    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg" pod "0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" (UID: "0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a"). InnerVolumeSpecName "kube-api-access-2ldcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:53.409043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:53.408864    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldcg\" (UniqueName: \"kubernetes.io/projected/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a-kube-api-access-2ldcg\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:54.360680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.351306    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:54.501699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.397228    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:31:54.501699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.397264    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:31:54.522328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.421508    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:31:54.560214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.560118    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:54.606206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.606163    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:54.620975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.615002    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:31:54.640990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.640879    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r" event={"ID":"8dd0e7ac-603f-4ef3-8209-93563d372317","Type":"ContainerDied","Data":"b77fb076fc9b0438a295ad2fd81ad1ab7a29a711b4bf5006a8250ee8835011a0"}
Oct 30 17:31:54.640990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.640945    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77fb076fc9b0438a295ad2fd81ad1ab7a29a711b4bf5006a8250ee8835011a0"
Oct 30 17:31:54.648678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.642503    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:54.861729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.837209    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 17:31:55.308210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:54.969748    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:55.308479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.022955    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca\") pod \"8dd0e7ac-603f-4ef3-8209-93563d372317\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") "
Oct 30 17:31:55.308624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.308602    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\") pod \"8dd0e7ac-603f-4ef3-8209-93563d372317\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") "
Oct 30 17:31:55.330668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.169076    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:31:55.331304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.278821    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:55.389669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.023363    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca" pod "8dd0e7ac-603f-4ef3-8209-93563d372317" (UID: "8dd0e7ac-603f-4ef3-8209-93563d372317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:55.491667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.490321    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:55.535375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.533317    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:55.561292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.561197    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:55.562256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.544572    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:31:55.588605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.544659    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:31:55.589286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.589260    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:31:55.589485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.589470    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:31:55.593662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.589668    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:55.636667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.545819    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:31:55.638823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.638799    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:55.674206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.640808    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:31:55.699813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.641947    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:31:55.700119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.672174    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca" pod "8dd0e7ac-603f-4ef3-8209-93563d372317" (UID: "8dd0e7ac-603f-4ef3-8209-93563d372317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:55.726864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.726830    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:55.727161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:55.727147    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:31:56.083947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.083919    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:31:56.143268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.136815    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:31:56.143268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.139612    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:31:56.143268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.140777    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd\" (UniqueName: \"kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd\") pod \"8dd0e7ac-603f-4ef3-8209-93563d372317\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") "
Oct 30 17:31:56.143268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.140939    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\") pod \"8dd0e7ac-603f-4ef3-8209-93563d372317\" (UID: \"8dd0e7ac-603f-4ef3-8209-93563d372317\") "
Oct 30 17:31:56.143268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.141831    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:56.161385 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:31:56.161247    5534 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8dd0e7ac-603f-4ef3-8209-93563d372317/volumes/kubernetes.io~secret/maas-api-xks-ca
Oct 30 17:31:56.161709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.161679    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca" pod "8dd0e7ac-603f-4ef3-8209-93563d372317" (UID: "8dd0e7ac-603f-4ef3-8209-93563d372317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:56.161992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.161970    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:31:56.275670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.264556    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd0e7ac-603f-4ef3-8209-93563d372317-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:56.304359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.304331    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:31:56.312742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.312704    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:31:56.313689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.313664    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:31:56.326661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.326289    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:31:56.337329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.336534    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:56.337499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.337481    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:31:56.347854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.347819    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd" pod "8dd0e7ac-603f-4ef3-8209-93563d372317" (UID: "8dd0e7ac-603f-4ef3-8209-93563d372317"). InnerVolumeSpecName "kube-api-access-2hwgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:56.348098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.348079    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"
Oct 30 17:31:56.368989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.368958    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwgd\" (UniqueName: \"kubernetes.io/projected/8dd0e7ac-603f-4ef3-8209-93563d372317-kube-api-access-2hwgd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:31:56.480101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:56.479423    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:31:57.130028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:57.129990    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:57.238992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:57.238950    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:58.030215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:58.030182    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:31:59.801206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:31:59.801139    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:01.482708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:01.482664    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:03.105345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:03.105292    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:04.927776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:04.927738    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:06.647705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:06.647669    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:08.802781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:08.802633    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:08.998262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:08.998203    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"]
Oct 30 17:32:08.999544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:08.999512    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" containerName="revoke-keys"
Oct 30 17:32:08.999731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:08.999714    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" containerName="revoke-keys"
Oct 30 17:32:08.999934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:08.999920    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8dd0e7ac-603f-4ef3-8209-93563d372317" containerName="revoke-keys"
Oct 30 17:32:09.000034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.000021    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd0e7ac-603f-4ef3-8209-93563d372317" containerName="revoke-keys"
Oct 30 17:32:09.000272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.000253    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd0e7ac-603f-4ef3-8209-93563d372317" containerName="revoke-keys"
Oct 30 17:32:09.000399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.000387    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" containerName="revoke-keys"
Oct 30 17:32:09.012042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.012019    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.019873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.019832    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"]
Oct 30 17:32:09.028751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.027829    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:09.028751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.027967    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:32:09.150893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.150850    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dll\" (UniqueName: \"kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.151151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.150937    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.151151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.151019    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.252381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.252335    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dll\" (UniqueName: \"kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.252578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.252401    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.252578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.252490    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.253529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.253500    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.255680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.255633    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.277684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.277618    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dll\" (UniqueName: \"kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll\") pod \"maas-api-revoke-sub-15de0f8e-72vlt\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.348665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.348036    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:09.613670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.611060    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"]
Oct 30 17:32:09.650990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.648935    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"]
Oct 30 17:32:09.650990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.649075    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.695373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.695337    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.695571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.695474    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.695571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.695507    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9zpj\" (UniqueName: \"kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.708490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.708147    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"]
Oct 30 17:32:09.796474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.796433    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.796677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.796580    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.796677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.796605    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9zpj\" (UniqueName: \"kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.797389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.797360    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.799089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.799064    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.820163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.820129    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9zpj\" (UniqueName: \"kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj\") pod \"maas-api-revoke-sub-a3f3239b-q7sh8\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:09.970983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:09.970952    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:10.277170 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:10.277139    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c4ad86_350d_41a1_aa26_561fa6f6167e.slice/crio-7a14ba1195f526f4214b64d494fd575b980980439a25d8bd13bf7f95493f7b0f WatchSource:0}: Error finding container 7a14ba1195f526f4214b64d494fd575b980980439a25d8bd13bf7f95493f7b0f: Status 404 returned error can't find the container with id 7a14ba1195f526f4214b64d494fd575b980980439a25d8bd13bf7f95493f7b0f
Oct 30 17:32:10.279953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:10.279922    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"]
Oct 30 17:32:10.547044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:10.546926    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt" event={"ID":"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c","Type":"ContainerStarted","Data":"08227bf1c18a2422bfb9a7dd3090a7640551b2ef4cdf4ff5b49a515c5fefd014"}
Oct 30 17:32:10.547235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:10.547043    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt" event={"ID":"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c","Type":"ContainerStarted","Data":"6ad204d9a6ba28a8d65b938b56d986546be8acdbdbc1275bfd4d36fde7b8a6fb"}
Oct 30 17:32:10.548377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:10.548352    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8" event={"ID":"b2c4ad86-350d-41a1-aa26-561fa6f6167e","Type":"ContainerStarted","Data":"7a14ba1195f526f4214b64d494fd575b980980439a25d8bd13bf7f95493f7b0f"}
Oct 30 17:32:10.589369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:10.589301    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt" podStartSLOduration=2.589282177 podStartE2EDuration="2.589282177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:10.587156469 +0000 UTC m=+1136.349200283" watchObservedRunningTime="2026-10-30 17:32:10.589282177 +0000 UTC m=+1136.351325990"
Oct 30 17:32:10.719114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:10.719067    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:11.366459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.366423    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"]
Oct 30 17:32:11.370983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.370962    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.392832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.392801    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"]
Oct 30 17:32:11.419954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.419920    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.420131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.420041    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqkk\" (UniqueName: \"kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.420254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.420235    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.521442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.521353    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.521586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.521473    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.521586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.521528    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqkk\" (UniqueName: \"kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.522241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.522216    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.523913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.523889    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.544603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.544578    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqkk\" (UniqueName: \"kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk\") pod \"maas-api-revoke-sub-6644fcd4-b6zhh\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.558874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.558835    5534 generic.go:356] "Generic (PLEG): container finished" podID="c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" containerID="08227bf1c18a2422bfb9a7dd3090a7640551b2ef4cdf4ff5b49a515c5fefd014" exitCode=0
Oct 30 17:32:11.559034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.558913    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt" event={"ID":"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c","Type":"ContainerDied","Data":"08227bf1c18a2422bfb9a7dd3090a7640551b2ef4cdf4ff5b49a515c5fefd014"}
Oct 30 17:32:11.560942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.560918    5534 generic.go:356] "Generic (PLEG): container finished" podID="b2c4ad86-350d-41a1-aa26-561fa6f6167e" containerID="42e952c79aeee0dcaf2297389059a6b4eb32e9ffeaabd51a637d8a9d334ee48d" exitCode=0
Oct 30 17:32:11.561068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.560989    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8" event={"ID":"b2c4ad86-350d-41a1-aa26-561fa6f6167e","Type":"ContainerDied","Data":"42e952c79aeee0dcaf2297389059a6b4eb32e9ffeaabd51a637d8a9d334ee48d"}
Oct 30 17:32:11.685504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.685460    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:11.998158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:11.998123    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"]
Oct 30 17:32:11.998934 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:11.998908    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f489f6a_7671_4e5a_9b84_f8d9344fd718.slice/crio-959bd521d060435f0e9450f57080dcce3a63e04d6ce2018e3a35593cd8244ce4 WatchSource:0}: Error finding container 959bd521d060435f0e9450f57080dcce3a63e04d6ce2018e3a35593cd8244ce4: Status 404 returned error can't find the container with id 959bd521d060435f0e9450f57080dcce3a63e04d6ce2018e3a35593cd8244ce4
Oct 30 17:32:12.364990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.364948    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:12.570902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.570806    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh" event={"ID":"3f489f6a-7671-4e5a-9b84-f8d9344fd718","Type":"ContainerStarted","Data":"6a45e280a00475f6e0c95bd956c4dbb7c4ca23b263f5f099cf3d812ec23c0d87"}
Oct 30 17:32:12.570902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.570878    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh" event={"ID":"3f489f6a-7671-4e5a-9b84-f8d9344fd718","Type":"ContainerStarted","Data":"959bd521d060435f0e9450f57080dcce3a63e04d6ce2018e3a35593cd8244ce4"}
Oct 30 17:32:12.612166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.612114    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh" podStartSLOduration=1.612098123 podStartE2EDuration="1.612098123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:12.609574217 +0000 UTC m=+1138.371618032" watchObservedRunningTime="2026-10-30 17:32:12.612098123 +0000 UTC m=+1138.374141927"
Oct 30 17:32:12.784914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.784890    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:12.816055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.816027    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:12.840354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.840328    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca\") pod \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") "
Oct 30 17:32:12.840525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.840402    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj\" (UniqueName: \"kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj\") pod \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") "
Oct 30 17:32:12.840525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.840513    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll\" (UniqueName: \"kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll\") pod \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") "
Oct 30 17:32:12.840616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.840546    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca\") pod \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\" (UID: \"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c\") "
Oct 30 17:32:12.840692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.840674    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca\") pod \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") "
Oct 30 17:32:12.840766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.840750    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca\") pod \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\" (UID: \"b2c4ad86-350d-41a1-aa26-561fa6f6167e\") "
Oct 30 17:32:12.841493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.841469    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca" pod "b2c4ad86-350d-41a1-aa26-561fa6f6167e" (UID: "b2c4ad86-350d-41a1-aa26-561fa6f6167e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:12.841778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.841758    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca" pod "c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" (UID: "c1208ff7-e011-4ceb-83f1-0e269b9c5e9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:12.846121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.845836    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll" pod "c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" (UID: "c1208ff7-e011-4ceb-83f1-0e269b9c5e9c"). InnerVolumeSpecName "kube-api-access-s4dll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:12.847024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.846847    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca" pod "b2c4ad86-350d-41a1-aa26-561fa6f6167e" (UID: "b2c4ad86-350d-41a1-aa26-561fa6f6167e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:12.851991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.851967    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj" pod "b2c4ad86-350d-41a1-aa26-561fa6f6167e" (UID: "b2c4ad86-350d-41a1-aa26-561fa6f6167e"). InnerVolumeSpecName "kube-api-access-z9zpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:12.853828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.853759    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca" pod "c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" (UID: "c1208ff7-e011-4ceb-83f1-0e269b9c5e9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:12.942954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.942919    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:12.942954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.942955    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c4ad86-350d-41a1-aa26-561fa6f6167e-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:12.943169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.942970    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:12.943169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.942985    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9zpj\" (UniqueName: \"kubernetes.io/projected/b2c4ad86-350d-41a1-aa26-561fa6f6167e-kube-api-access-z9zpj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:12.943169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.942999    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4dll\" (UniqueName: \"kubernetes.io/projected/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-kube-api-access-s4dll\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:12.943169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:12.943013    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:13.581075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.581044    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"
Oct 30 17:32:13.581075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.581056    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt" event={"ID":"c1208ff7-e011-4ceb-83f1-0e269b9c5e9c","Type":"ContainerDied","Data":"6ad204d9a6ba28a8d65b938b56d986546be8acdbdbc1275bfd4d36fde7b8a6fb"}
Oct 30 17:32:13.581607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.581115    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ad204d9a6ba28a8d65b938b56d986546be8acdbdbc1275bfd4d36fde7b8a6fb"
Oct 30 17:32:13.583425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.583405    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"
Oct 30 17:32:13.583618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.583436    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8" event={"ID":"b2c4ad86-350d-41a1-aa26-561fa6f6167e","Type":"ContainerDied","Data":"7a14ba1195f526f4214b64d494fd575b980980439a25d8bd13bf7f95493f7b0f"}
Oct 30 17:32:13.583618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.583483    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a14ba1195f526f4214b64d494fd575b980980439a25d8bd13bf7f95493f7b0f"
Oct 30 17:32:13.585622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.585593    5534 generic.go:356] "Generic (PLEG): container finished" podID="3f489f6a-7671-4e5a-9b84-f8d9344fd718" containerID="6a45e280a00475f6e0c95bd956c4dbb7c4ca23b263f5f099cf3d812ec23c0d87" exitCode=0
Oct 30 17:32:13.585767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:13.585712    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh" event={"ID":"3f489f6a-7671-4e5a-9b84-f8d9344fd718","Type":"ContainerDied","Data":"6a45e280a00475f6e0c95bd956c4dbb7c4ca23b263f5f099cf3d812ec23c0d87"}
Oct 30 17:32:14.078383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.078340    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:14.767870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.767841    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:14.872943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.872848    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca\") pod \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") "
Oct 30 17:32:14.873106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.873043    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk\" (UniqueName: \"kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk\") pod \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") "
Oct 30 17:32:14.873106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.873077    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca\") pod \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\" (UID: \"3f489f6a-7671-4e5a-9b84-f8d9344fd718\") "
Oct 30 17:32:14.873248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.873212    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca" pod "3f489f6a-7671-4e5a-9b84-f8d9344fd718" (UID: "3f489f6a-7671-4e5a-9b84-f8d9344fd718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:14.873525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.873500    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:14.875416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.875388    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca" pod "3f489f6a-7671-4e5a-9b84-f8d9344fd718" (UID: "3f489f6a-7671-4e5a-9b84-f8d9344fd718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:14.875603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.875578    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk" pod "3f489f6a-7671-4e5a-9b84-f8d9344fd718" (UID: "3f489f6a-7671-4e5a-9b84-f8d9344fd718"). InnerVolumeSpecName "kube-api-access-hxqkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:14.975512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.975471    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqkk\" (UniqueName: \"kubernetes.io/projected/3f489f6a-7671-4e5a-9b84-f8d9344fd718-kube-api-access-hxqkk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:14.975512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:14.975511    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f489f6a-7671-4e5a-9b84-f8d9344fd718-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:15.243917 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:32:15.243833    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:32:15.605765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:15.605734    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"
Oct 30 17:32:15.605972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:15.605736    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh" event={"ID":"3f489f6a-7671-4e5a-9b84-f8d9344fd718","Type":"ContainerDied","Data":"959bd521d060435f0e9450f57080dcce3a63e04d6ce2018e3a35593cd8244ce4"}
Oct 30 17:32:15.605972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:15.605862    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="959bd521d060435f0e9450f57080dcce3a63e04d6ce2018e3a35593cd8244ce4"
Oct 30 17:32:15.679661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:15.679609    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:17.424717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:17.424680    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:19.008365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.008327    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:19.975311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.975272    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"]
Oct 30 17:32:19.976121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976098    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" containerName="revoke-keys"
Oct 30 17:32:19.976121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976121    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" containerName="revoke-keys"
Oct 30 17:32:19.976329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976177    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3f489f6a-7671-4e5a-9b84-f8d9344fd718" containerName="revoke-keys"
Oct 30 17:32:19.976329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976187    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f489f6a-7671-4e5a-9b84-f8d9344fd718" containerName="revoke-keys"
Oct 30 17:32:19.976329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976203    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b2c4ad86-350d-41a1-aa26-561fa6f6167e" containerName="revoke-keys"
Oct 30 17:32:19.976329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976210    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c4ad86-350d-41a1-aa26-561fa6f6167e" containerName="revoke-keys"
Oct 30 17:32:19.976525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976378    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c4ad86-350d-41a1-aa26-561fa6f6167e" containerName="revoke-keys"
Oct 30 17:32:19.976525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976401    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" containerName="revoke-keys"
Oct 30 17:32:19.976525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.976418    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f489f6a-7671-4e5a-9b84-f8d9344fd718" containerName="revoke-keys"
Oct 30 17:32:19.979886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.979866    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:19.994979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:19.994952    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:20.007158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.007129    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:32:20.012014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.011982    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"]
Oct 30 17:32:20.052349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.052319    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.052542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.052412    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.052542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.052472    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gsq\" (UniqueName: \"kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.154447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.154412    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.154692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.154470    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gsq\" (UniqueName: \"kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.154781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.154730    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.155387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.155364    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.157030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.157010    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.177729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.177698    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gsq\" (UniqueName: \"kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq\") pod \"maas-api-revoke-sub-e26f3d67-dzqnk\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.294854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.294751    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:20.607251 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:20.607210    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532b81fe_161a_4f80_bd45_ffd994fb2941.slice/crio-f056a1a65135edf217d3d887677335b18972ea224034c47acec93f19fdcbcfc0 WatchSource:0}: Error finding container f056a1a65135edf217d3d887677335b18972ea224034c47acec93f19fdcbcfc0: Status 404 returned error can't find the container with id f056a1a65135edf217d3d887677335b18972ea224034c47acec93f19fdcbcfc0
Oct 30 17:32:20.608379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.608349    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"]
Oct 30 17:32:20.623105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.623063    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"]
Oct 30 17:32:20.643302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.643278    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.646731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.646700    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"]
Oct 30 17:32:20.658550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.650206    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk" event={"ID":"532b81fe-161a-4f80-bd45-ffd994fb2941","Type":"ContainerStarted","Data":"f056a1a65135edf217d3d887677335b18972ea224034c47acec93f19fdcbcfc0"}
Oct 30 17:32:20.764924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.764894    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbj65\" (UniqueName: \"kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.765163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.765143    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.765208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.765173    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.866346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.866263    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.866346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.866304    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.866562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.866427    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbj65\" (UniqueName: \"kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.867082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.866998    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.868782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.868761    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.890170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.890144    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbj65\" (UniqueName: \"kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65\") pod \"maas-api-revoke-sub-8caa8ce0-scc6g\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:20.907969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.907942    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:20.972954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:20.972916    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:21.299258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:21.299217    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"]
Oct 30 17:32:21.660080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:21.660042    5534 generic.go:356] "Generic (PLEG): container finished" podID="532b81fe-161a-4f80-bd45-ffd994fb2941" containerID="8bbee5bb38d58bdf0e16ee3549a536f4e4215e3d7a109e5b12145f81c0e995af" exitCode=0
Oct 30 17:32:21.660295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:21.660141    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk" event={"ID":"532b81fe-161a-4f80-bd45-ffd994fb2941","Type":"ContainerDied","Data":"8bbee5bb38d58bdf0e16ee3549a536f4e4215e3d7a109e5b12145f81c0e995af"}
Oct 30 17:32:21.662114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:21.662090    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g" event={"ID":"135bc2e4-6bd5-46a2-b7bc-c13c429e5731","Type":"ContainerStarted","Data":"c72a079a6c2547475297ce3ae80e7193c7da8e40b830432e8f9b9dcc927a90f9"}
Oct 30 17:32:21.662232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:21.662133    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g" event={"ID":"135bc2e4-6bd5-46a2-b7bc-c13c429e5731","Type":"ContainerStarted","Data":"d6e98bceda900392b9841334400cf03a624682acb180a0840cec7a8e3ca2ae93"}
Oct 30 17:32:21.738738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:21.738678    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g" podStartSLOduration=1.7386175879999999 podStartE2EDuration="1.738617588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:21.73821169 +0000 UTC m=+1147.500255503" watchObservedRunningTime="2026-10-30 17:32:21.738617588 +0000 UTC m=+1147.500661492"
Oct 30 17:32:22.630050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:22.630014    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:22.680390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:22.680349    5534 generic.go:356] "Generic (PLEG): container finished" podID="135bc2e4-6bd5-46a2-b7bc-c13c429e5731" containerID="c72a079a6c2547475297ce3ae80e7193c7da8e40b830432e8f9b9dcc927a90f9" exitCode=0
Oct 30 17:32:22.681695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:22.681628    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g" event={"ID":"135bc2e4-6bd5-46a2-b7bc-c13c429e5731","Type":"ContainerDied","Data":"c72a079a6c2547475297ce3ae80e7193c7da8e40b830432e8f9b9dcc927a90f9"}
Oct 30 17:32:22.907518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:22.907490    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:23.001416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.001376    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca\") pod \"532b81fe-161a-4f80-bd45-ffd994fb2941\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") "
Oct 30 17:32:23.001574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.001497    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq\" (UniqueName: \"kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq\") pod \"532b81fe-161a-4f80-bd45-ffd994fb2941\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") "
Oct 30 17:32:23.001574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.001559    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca\") pod \"532b81fe-161a-4f80-bd45-ffd994fb2941\" (UID: \"532b81fe-161a-4f80-bd45-ffd994fb2941\") "
Oct 30 17:32:23.001975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.001943    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca" pod "532b81fe-161a-4f80-bd45-ffd994fb2941" (UID: "532b81fe-161a-4f80-bd45-ffd994fb2941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:23.002340 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.002314    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:23.003823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.003796    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca" pod "532b81fe-161a-4f80-bd45-ffd994fb2941" (UID: "532b81fe-161a-4f80-bd45-ffd994fb2941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:23.003922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.003847    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq" pod "532b81fe-161a-4f80-bd45-ffd994fb2941" (UID: "532b81fe-161a-4f80-bd45-ffd994fb2941"). InnerVolumeSpecName "kube-api-access-f5gsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:23.103794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.103756    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/532b81fe-161a-4f80-bd45-ffd994fb2941-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:23.103794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.103785    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5gsq\" (UniqueName: \"kubernetes.io/projected/532b81fe-161a-4f80-bd45-ffd994fb2941-kube-api-access-f5gsq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:23.627126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.627095    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"]
Oct 30 17:32:23.627741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.627725    5534 cpu_manager.go:414] "containerMap: removing container" podUID="532b81fe-161a-4f80-bd45-ffd994fb2941" containerName="revoke-keys"
Oct 30 17:32:23.627804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.627743    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532b81fe-161a-4f80-bd45-ffd994fb2941" containerName="revoke-keys"
Oct 30 17:32:23.627914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.627904    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="532b81fe-161a-4f80-bd45-ffd994fb2941" containerName="revoke-keys"
Oct 30 17:32:23.631618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.631598    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.651357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.651326    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"]
Oct 30 17:32:23.699126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.698943    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk" event={"ID":"532b81fe-161a-4f80-bd45-ffd994fb2941","Type":"ContainerDied","Data":"f056a1a65135edf217d3d887677335b18972ea224034c47acec93f19fdcbcfc0"}
Oct 30 17:32:23.699126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.699037    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f056a1a65135edf217d3d887677335b18972ea224034c47acec93f19fdcbcfc0"
Oct 30 17:32:23.700267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.700235    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"
Oct 30 17:32:23.814662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.814235    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.814662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.814289    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4c7s\" (UniqueName: \"kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.814662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.814389    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.868468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.868428    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:23.915834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.915757    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.915834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.915795    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c7s\" (UniqueName: \"kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.916072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.915879    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.916621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.916539    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.918746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.918697    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.940010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.939977    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c7s\" (UniqueName: \"kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s\") pod \"maas-api-revoke-sub-f233b398-qp95f\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:23.945533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:23.945507    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:24.017347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.017314    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca\") pod \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") "
Oct 30 17:32:24.017554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.017515    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca\") pod \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") "
Oct 30 17:32:24.017631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.017614    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65\" (UniqueName: \"kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65\") pod \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\" (UID: \"135bc2e4-6bd5-46a2-b7bc-c13c429e5731\") "
Oct 30 17:32:24.018690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.018457    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca" pod "135bc2e4-6bd5-46a2-b7bc-c13c429e5731" (UID: "135bc2e4-6bd5-46a2-b7bc-c13c429e5731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:24.020169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.019592    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:24.021335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.021306    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca" pod "135bc2e4-6bd5-46a2-b7bc-c13c429e5731" (UID: "135bc2e4-6bd5-46a2-b7bc-c13c429e5731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:24.021467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.021335    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65" pod "135bc2e4-6bd5-46a2-b7bc-c13c429e5731" (UID: "135bc2e4-6bd5-46a2-b7bc-c13c429e5731"). InnerVolumeSpecName "kube-api-access-lbj65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:24.095664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.095491    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"]
Oct 30 17:32:24.098906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.096578    5534 cpu_manager.go:414] "containerMap: removing container" podUID="135bc2e4-6bd5-46a2-b7bc-c13c429e5731" containerName="revoke-keys"
Oct 30 17:32:24.098906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.096604    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135bc2e4-6bd5-46a2-b7bc-c13c429e5731" containerName="revoke-keys"
Oct 30 17:32:24.098906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.096776    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="135bc2e4-6bd5-46a2-b7bc-c13c429e5731" containerName="revoke-keys"
Oct 30 17:32:24.102618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.102596    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.110679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.109716    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"]
Oct 30 17:32:24.120699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.120675    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:24.120779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.120705    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbj65\" (UniqueName: \"kubernetes.io/projected/135bc2e4-6bd5-46a2-b7bc-c13c429e5731-kube-api-access-lbj65\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:24.221750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.221672    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.221750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.221746    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.221965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.221842    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qvk\" (UniqueName: \"kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.272160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.272129    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"]
Oct 30 17:32:24.323061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.323027    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qvk\" (UniqueName: \"kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.323338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.323319    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.323390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.323364    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.323969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.323942    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.325738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.325720    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.346413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.346392    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qvk\" (UniqueName: \"kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk\") pod \"maas-api-revoke-sub-aab573e5-fsn5q\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.421405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.421367    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:24.537922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.537880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:24.709397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.709364    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"
Oct 30 17:32:24.709397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.709384    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g" event={"ID":"135bc2e4-6bd5-46a2-b7bc-c13c429e5731","Type":"ContainerDied","Data":"d6e98bceda900392b9841334400cf03a624682acb180a0840cec7a8e3ca2ae93"}
Oct 30 17:32:24.709941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.709433    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e98bceda900392b9841334400cf03a624682acb180a0840cec7a8e3ca2ae93"
Oct 30 17:32:24.710870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.710844    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f" event={"ID":"c0c8a1ef-b710-4c57-b1c9-7a721202f156","Type":"ContainerStarted","Data":"7c7ab5ead4cb90488e3708cc5223cf3b2fc01b88aebf4a6c3218a18bce00cb41"}
Oct 30 17:32:24.742092 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:24.742066    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dcc52e_6e49_4eb5_baef_f3f589159bc7.slice/crio-cfca5625cfc654a61f175e509cbe420856afbad7753056ce2d03a15c4ee216ce WatchSource:0}: Error finding container cfca5625cfc654a61f175e509cbe420856afbad7753056ce2d03a15c4ee216ce: Status 404 returned error can't find the container with id cfca5625cfc654a61f175e509cbe420856afbad7753056ce2d03a15c4ee216ce
Oct 30 17:32:24.744802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:24.744778    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"]
Oct 30 17:32:25.719803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:25.719769    5534 generic.go:356] "Generic (PLEG): container finished" podID="67dcc52e-6e49-4eb5-baef-f3f589159bc7" containerID="b77778a8c459e2333128cc394c7cab4904d7e9e6b646b1b7fd5db8e6b4cece2e" exitCode=0
Oct 30 17:32:25.720221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:25.719846    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q" event={"ID":"67dcc52e-6e49-4eb5-baef-f3f589159bc7","Type":"ContainerDied","Data":"b77778a8c459e2333128cc394c7cab4904d7e9e6b646b1b7fd5db8e6b4cece2e"}
Oct 30 17:32:25.720221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:25.719903    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q" event={"ID":"67dcc52e-6e49-4eb5-baef-f3f589159bc7","Type":"ContainerStarted","Data":"cfca5625cfc654a61f175e509cbe420856afbad7753056ce2d03a15c4ee216ce"}
Oct 30 17:32:25.721660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:25.721618    5534 generic.go:356] "Generic (PLEG): container finished" podID="c0c8a1ef-b710-4c57-b1c9-7a721202f156" containerID="e3a7136ac6029147536fffc837527b2952fd29f7c189eba61a46f538e543d1c3" exitCode=0
Oct 30 17:32:25.721798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:25.721689    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f" event={"ID":"c0c8a1ef-b710-4c57-b1c9-7a721202f156","Type":"ContainerDied","Data":"e3a7136ac6029147536fffc837527b2952fd29f7c189eba61a46f538e543d1c3"}
Oct 30 17:32:26.196390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.196357    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:26.310770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.310736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:26.931836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.931809    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:26.963161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.963132    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca\") pod \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") "
Oct 30 17:32:26.963351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.963183    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca\") pod \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") "
Oct 30 17:32:26.963411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.963371    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk\" (UniqueName: \"kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk\") pod \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\" (UID: \"67dcc52e-6e49-4eb5-baef-f3f589159bc7\") "
Oct 30 17:32:26.963895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.963869    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca" pod "67dcc52e-6e49-4eb5-baef-f3f589159bc7" (UID: "67dcc52e-6e49-4eb5-baef-f3f589159bc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:26.964180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.964157    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:26.966108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.966083    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk" pod "67dcc52e-6e49-4eb5-baef-f3f589159bc7" (UID: "67dcc52e-6e49-4eb5-baef-f3f589159bc7"). InnerVolumeSpecName "kube-api-access-67qvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:26.966230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.966204    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca" pod "67dcc52e-6e49-4eb5-baef-f3f589159bc7" (UID: "67dcc52e-6e49-4eb5-baef-f3f589159bc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:26.991693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:26.991573    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:27.065371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.065331    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca\") pod \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") "
Oct 30 17:32:27.065541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.065402    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s\" (UniqueName: \"kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s\") pod \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") "
Oct 30 17:32:27.066635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.065754    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca\") pod \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\" (UID: \"c0c8a1ef-b710-4c57-b1c9-7a721202f156\") "
Oct 30 17:32:27.066635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.065848    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca" pod "c0c8a1ef-b710-4c57-b1c9-7a721202f156" (UID: "c0c8a1ef-b710-4c57-b1c9-7a721202f156"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:27.066635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.066558    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dcc52e-6e49-4eb5-baef-f3f589159bc7-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:27.066635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.066582    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:27.066635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.066597    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qvk\" (UniqueName: \"kubernetes.io/projected/67dcc52e-6e49-4eb5-baef-f3f589159bc7-kube-api-access-67qvk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:27.068581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.068550    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s" pod "c0c8a1ef-b710-4c57-b1c9-7a721202f156" (UID: "c0c8a1ef-b710-4c57-b1c9-7a721202f156"). InnerVolumeSpecName "kube-api-access-s4c7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:27.068734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.068704    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca" pod "c0c8a1ef-b710-4c57-b1c9-7a721202f156" (UID: "c0c8a1ef-b710-4c57-b1c9-7a721202f156"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:27.168100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.168063    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4c7s\" (UniqueName: \"kubernetes.io/projected/c0c8a1ef-b710-4c57-b1c9-7a721202f156-kube-api-access-s4c7s\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:27.168100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.168098    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c8a1ef-b710-4c57-b1c9-7a721202f156-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:27.739347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.739316    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"
Oct 30 17:32:27.739542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.739345    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q" event={"ID":"67dcc52e-6e49-4eb5-baef-f3f589159bc7","Type":"ContainerDied","Data":"cfca5625cfc654a61f175e509cbe420856afbad7753056ce2d03a15c4ee216ce"}
Oct 30 17:32:27.739542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.739402    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfca5625cfc654a61f175e509cbe420856afbad7753056ce2d03a15c4ee216ce"
Oct 30 17:32:27.741024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.740991    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f" event={"ID":"c0c8a1ef-b710-4c57-b1c9-7a721202f156","Type":"ContainerDied","Data":"7c7ab5ead4cb90488e3708cc5223cf3b2fc01b88aebf4a6c3218a18bce00cb41"}
Oct 30 17:32:27.741165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.741054    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7ab5ead4cb90488e3708cc5223cf3b2fc01b88aebf4a6c3218a18bce00cb41"
Oct 30 17:32:27.741165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:27.741003    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"
Oct 30 17:32:28.026612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:28.026521    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:29.226038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.225992    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:29.267253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.267216    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"]
Oct 30 17:32:29.268193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.268167    5534 cpu_manager.go:414] "containerMap: removing container" podUID="67dcc52e-6e49-4eb5-baef-f3f589159bc7" containerName="revoke-keys"
Oct 30 17:32:29.268193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.268191    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dcc52e-6e49-4eb5-baef-f3f589159bc7" containerName="revoke-keys"
Oct 30 17:32:29.268369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.268243    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c0c8a1ef-b710-4c57-b1c9-7a721202f156" containerName="revoke-keys"
Oct 30 17:32:29.268369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.268251    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c8a1ef-b710-4c57-b1c9-7a721202f156" containerName="revoke-keys"
Oct 30 17:32:29.268516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.268436    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dcc52e-6e49-4eb5-baef-f3f589159bc7" containerName="revoke-keys"
Oct 30 17:32:29.268516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.268457    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c8a1ef-b710-4c57-b1c9-7a721202f156" containerName="revoke-keys"
Oct 30 17:32:29.274702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.274674    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.309982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.309950    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"]
Oct 30 17:32:29.311747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.311719    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.312089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.312066    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.312274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.312259    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmdx\" (UniqueName: \"kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.316560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.316538    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:32:29.316847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.316831    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:29.413685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.413631    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.414021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.413998    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.414128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.414103    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmdx\" (UniqueName: \"kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.414558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.414528    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.416883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.416857    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.438027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.438006    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmdx\" (UniqueName: \"kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx\") pod \"maas-api-revoke-sub-41f217ac-wtwr7\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.668234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.667911    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:29.946816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:29.946683    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:30.828903 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:30.828863    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdf13e4_092a_456e_9f44_180f3066103b.slice/crio-fb51b0bfe42db5ebf20362ae11fd6a6784f5303d25d57d8b15aab8fe5e27697c WatchSource:0}: Error finding container fb51b0bfe42db5ebf20362ae11fd6a6784f5303d25d57d8b15aab8fe5e27697c: Status 404 returned error can't find the container with id fb51b0bfe42db5ebf20362ae11fd6a6784f5303d25d57d8b15aab8fe5e27697c
Oct 30 17:32:30.917337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:30.916537    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"]
Oct 30 17:32:31.158682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.137739    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.162563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.162523    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.313898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.313849    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.327226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.327178    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.370034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.366748    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:32:31.440758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.437211    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.511575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.511529    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.634748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.634513    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.846679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.846462    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:31.941464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:31.941405    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.047690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.047576    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7" event={"ID":"5fdf13e4-092a-456e-9f44-180f3066103b","Type":"ContainerStarted","Data":"7836c5932e26002a96239d25dfab346e178043569e5d6c967d42ab13e36a5346"}
Oct 30 17:32:32.047993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.047955    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7" event={"ID":"5fdf13e4-092a-456e-9f44-180f3066103b","Type":"ContainerStarted","Data":"fb51b0bfe42db5ebf20362ae11fd6a6784f5303d25d57d8b15aab8fe5e27697c"}
Oct 30 17:32:32.079823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.079790    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:32:32.080056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.080032    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.166848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.166791    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.184840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.172243    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.184840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.172604    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:32:32.189975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.188738    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.189975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.189109    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:32:32.333288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.332797    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:32:32.333288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.332866    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.333741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.333687    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.334336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.334318    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:32:32.388924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.388882    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:32:32.389136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.388952    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.445319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.444900    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.559435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.484065    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7" podStartSLOduration=3.484041318 podStartE2EDuration="3.484041318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:32.09489623 +0000 UTC m=+1157.856940044" watchObservedRunningTime="2026-10-30 17:32:32.484041318 +0000 UTC m=+1158.246085132"
Oct 30 17:32:32.821497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.821414    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.912667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.854479    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:32.912667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.886970    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:32.993672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.991053    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:32:32.993672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.991116    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:32:32.993672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.991148    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:32:32.993672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.991180    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:32:32.993672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:32.991220    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:32:33.023719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:33.023213    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:32:33.057240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:33.057212    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:32:33.097425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:33.092967    5534 generic.go:356] "Generic (PLEG): container finished" podID="5fdf13e4-092a-456e-9f44-180f3066103b" containerID="7836c5932e26002a96239d25dfab346e178043569e5d6c967d42ab13e36a5346" exitCode=0
Oct 30 17:32:33.097425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:33.094013    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7" event={"ID":"5fdf13e4-092a-456e-9f44-180f3066103b","Type":"ContainerDied","Data":"7836c5932e26002a96239d25dfab346e178043569e5d6c967d42ab13e36a5346"}
Oct 30 17:32:34.278379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.278345    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:34.367108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.367018    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca\") pod \"5fdf13e4-092a-456e-9f44-180f3066103b\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") "
Oct 30 17:32:34.367108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.367086    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca\") pod \"5fdf13e4-092a-456e-9f44-180f3066103b\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") "
Oct 30 17:32:34.367337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.367262    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx\" (UniqueName: \"kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx\") pod \"5fdf13e4-092a-456e-9f44-180f3066103b\" (UID: \"5fdf13e4-092a-456e-9f44-180f3066103b\") "
Oct 30 17:32:34.367611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.367480    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca" pod "5fdf13e4-092a-456e-9f44-180f3066103b" (UID: "5fdf13e4-092a-456e-9f44-180f3066103b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:34.368038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.368013    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:34.369841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.369815    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca" pod "5fdf13e4-092a-456e-9f44-180f3066103b" (UID: "5fdf13e4-092a-456e-9f44-180f3066103b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:34.370013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.369993    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx" pod "5fdf13e4-092a-456e-9f44-180f3066103b" (UID: "5fdf13e4-092a-456e-9f44-180f3066103b"). InnerVolumeSpecName "kube-api-access-fpmdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:34.469766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.469728    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdf13e4-092a-456e-9f44-180f3066103b-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:34.469766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.469765    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmdx\" (UniqueName: \"kubernetes.io/projected/5fdf13e4-092a-456e-9f44-180f3066103b-kube-api-access-fpmdx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:34.504447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:34.504414    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:35.118996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:35.118970    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"
Oct 30 17:32:35.118996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:35.118983    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7" event={"ID":"5fdf13e4-092a-456e-9f44-180f3066103b","Type":"ContainerDied","Data":"fb51b0bfe42db5ebf20362ae11fd6a6784f5303d25d57d8b15aab8fe5e27697c"}
Oct 30 17:32:35.119210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:35.119039    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb51b0bfe42db5ebf20362ae11fd6a6784f5303d25d57d8b15aab8fe5e27697c"
Oct 30 17:32:36.280806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.280760    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:36.607998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.607955    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"]
Oct 30 17:32:36.608636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.608614    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5fdf13e4-092a-456e-9f44-180f3066103b" containerName="revoke-keys"
Oct 30 17:32:36.608636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.608634    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdf13e4-092a-456e-9f44-180f3066103b" containerName="revoke-keys"
Oct 30 17:32:36.608879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.608833    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdf13e4-092a-456e-9f44-180f3066103b" containerName="revoke-keys"
Oct 30 17:32:36.612856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.612833    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.633078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.633046    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"]
Oct 30 17:32:36.645671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.638423    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:36.645671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.638499    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:32:36.702655 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.702591    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.702874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.702850    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9ts\" (UniqueName: \"kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.702969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.702952    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.817234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.817189    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9ts\" (UniqueName: \"kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.817426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.817323    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.817474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.817450    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.818108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.818070    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.819927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.819906    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.840156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.840121    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9ts\" (UniqueName: \"kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts\") pod \"maas-api-revoke-sub-1d250889-lzs47\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:36.929477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:36.929391    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:37.255910 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:37.255875    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5859cd_3152_4a76_bd22_47ae1f877556.slice/crio-a664d16df4c54a66fa66d81594cae7ae50e3e99b9908eda3c83126a7d3f659f3 WatchSource:0}: Error finding container a664d16df4c54a66fa66d81594cae7ae50e3e99b9908eda3c83126a7d3f659f3: Status 404 returned error can't find the container with id a664d16df4c54a66fa66d81594cae7ae50e3e99b9908eda3c83126a7d3f659f3
Oct 30 17:32:37.256215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:37.256196    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"]
Oct 30 17:32:38.089912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.089876    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:38.154327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.154271    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47" event={"ID":"cd5859cd-3152-4a76-bd22-47ae1f877556","Type":"ContainerStarted","Data":"91b53e7cafb716ef06da2bfdd3d779816d09ea80e9b4f66d3075f51950341033"}
Oct 30 17:32:38.154500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.154359    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47" event={"ID":"cd5859cd-3152-4a76-bd22-47ae1f877556","Type":"ContainerStarted","Data":"a664d16df4c54a66fa66d81594cae7ae50e3e99b9908eda3c83126a7d3f659f3"}
Oct 30 17:32:38.198708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.198636    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47" podStartSLOduration=2.198612505 podStartE2EDuration="2.198612505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:38.194949328 +0000 UTC m=+1163.956993142" watchObservedRunningTime="2026-10-30 17:32:38.198612505 +0000 UTC m=+1163.960656318"
Oct 30 17:32:38.885447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.885417    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"]
Oct 30 17:32:38.890797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.890773    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:38.909117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:38.909083    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"]
Oct 30 17:32:39.054780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.054743    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.055152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.055123    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7x6\" (UniqueName: \"kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.055297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.055264    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.157271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.157172    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.157271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.157219    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.157831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.157454    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7x6\" (UniqueName: \"kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.158114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.158086    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.160308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.160282    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.166558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.166530    5534 generic.go:356] "Generic (PLEG): container finished" podID="cd5859cd-3152-4a76-bd22-47ae1f877556" containerID="91b53e7cafb716ef06da2bfdd3d779816d09ea80e9b4f66d3075f51950341033" exitCode=0
Oct 30 17:32:39.166750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.166610    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47" event={"ID":"cd5859cd-3152-4a76-bd22-47ae1f877556","Type":"ContainerDied","Data":"91b53e7cafb716ef06da2bfdd3d779816d09ea80e9b4f66d3075f51950341033"}
Oct 30 17:32:39.180775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.180746    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7x6\" (UniqueName: \"kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6\") pod \"maas-api-revoke-sub-490b2856-5k972\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.216021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.215979    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:39.232288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.232258    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:39.528681 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:39.528292    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc4ea5a_d1e2_4007_91cc_53fb81272f97.slice/crio-f27cf40e7e52fac6119bbc3e9f36b32c8f84de5d88a9fea3ce08550e35e6d49f WatchSource:0}: Error finding container f27cf40e7e52fac6119bbc3e9f36b32c8f84de5d88a9fea3ce08550e35e6d49f: Status 404 returned error can't find the container with id f27cf40e7e52fac6119bbc3e9f36b32c8f84de5d88a9fea3ce08550e35e6d49f
Oct 30 17:32:39.531774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.531746    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"]
Oct 30 17:32:39.816832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:39.816794    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:40.178944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.178910    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972" event={"ID":"2fc4ea5a-d1e2-4007-91cc-53fb81272f97","Type":"ContainerStarted","Data":"c0fa8c22f75feee05803df0f09c2c15d4ebcaefd23cd1bc9b7469cdc4eb9bdc8"}
Oct 30 17:32:40.179379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.179004    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972" event={"ID":"2fc4ea5a-d1e2-4007-91cc-53fb81272f97","Type":"ContainerStarted","Data":"f27cf40e7e52fac6119bbc3e9f36b32c8f84de5d88a9fea3ce08550e35e6d49f"}
Oct 30 17:32:40.218825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.218769    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972" podStartSLOduration=2.218753455 podStartE2EDuration="2.218753455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:40.217980746 +0000 UTC m=+1165.980024561" watchObservedRunningTime="2026-10-30 17:32:40.218753455 +0000 UTC m=+1165.980797291"
Oct 30 17:32:40.345172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.345146    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:40.479267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.479180    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts\" (UniqueName: \"kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts\") pod \"cd5859cd-3152-4a76-bd22-47ae1f877556\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") "
Oct 30 17:32:40.479484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.479457    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca\") pod \"cd5859cd-3152-4a76-bd22-47ae1f877556\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") "
Oct 30 17:32:40.479910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.479618    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca\") pod \"cd5859cd-3152-4a76-bd22-47ae1f877556\" (UID: \"cd5859cd-3152-4a76-bd22-47ae1f877556\") "
Oct 30 17:32:40.480378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.480345    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca" pod "cd5859cd-3152-4a76-bd22-47ae1f877556" (UID: "cd5859cd-3152-4a76-bd22-47ae1f877556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:40.481700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.481673    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts" pod "cd5859cd-3152-4a76-bd22-47ae1f877556" (UID: "cd5859cd-3152-4a76-bd22-47ae1f877556"). InnerVolumeSpecName "kube-api-access-7x9ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:40.481929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.481903    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca" pod "cd5859cd-3152-4a76-bd22-47ae1f877556" (UID: "cd5859cd-3152-4a76-bd22-47ae1f877556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:40.582250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.582214    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:40.582250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.582245    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9ts\" (UniqueName: \"kubernetes.io/projected/cd5859cd-3152-4a76-bd22-47ae1f877556-kube-api-access-7x9ts\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:40.582250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:40.582256    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd5859cd-3152-4a76-bd22-47ae1f877556-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:41.193422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:41.193388    5534 generic.go:356] "Generic (PLEG): container finished" podID="2fc4ea5a-d1e2-4007-91cc-53fb81272f97" containerID="c0fa8c22f75feee05803df0f09c2c15d4ebcaefd23cd1bc9b7469cdc4eb9bdc8" exitCode=0
Oct 30 17:32:41.193925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:41.193528    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972" event={"ID":"2fc4ea5a-d1e2-4007-91cc-53fb81272f97","Type":"ContainerDied","Data":"c0fa8c22f75feee05803df0f09c2c15d4ebcaefd23cd1bc9b7469cdc4eb9bdc8"}
Oct 30 17:32:41.195786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:41.195468    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"
Oct 30 17:32:41.195786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:41.195471    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47" event={"ID":"cd5859cd-3152-4a76-bd22-47ae1f877556","Type":"ContainerDied","Data":"a664d16df4c54a66fa66d81594cae7ae50e3e99b9908eda3c83126a7d3f659f3"}
Oct 30 17:32:41.195786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:41.195596    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a664d16df4c54a66fa66d81594cae7ae50e3e99b9908eda3c83126a7d3f659f3"
Oct 30 17:32:41.484687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:41.484582    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:42.374127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.374097    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:42.512203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.512118    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca\") pod \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") "
Oct 30 17:32:42.512342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.512271    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca\") pod \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") "
Oct 30 17:32:42.512342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.512338    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6\" (UniqueName: \"kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6\") pod \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\" (UID: \"2fc4ea5a-d1e2-4007-91cc-53fb81272f97\") "
Oct 30 17:32:42.512831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.512789    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca" pod "2fc4ea5a-d1e2-4007-91cc-53fb81272f97" (UID: "2fc4ea5a-d1e2-4007-91cc-53fb81272f97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:42.513560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.513538    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:42.514608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.514581    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca" pod "2fc4ea5a-d1e2-4007-91cc-53fb81272f97" (UID: "2fc4ea5a-d1e2-4007-91cc-53fb81272f97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:42.514740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.514668    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6" pod "2fc4ea5a-d1e2-4007-91cc-53fb81272f97" (UID: "2fc4ea5a-d1e2-4007-91cc-53fb81272f97"). InnerVolumeSpecName "kube-api-access-bj7x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:42.615087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.615054    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:42.615087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:42.615082    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7x6\" (UniqueName: \"kubernetes.io/projected/2fc4ea5a-d1e2-4007-91cc-53fb81272f97-kube-api-access-bj7x6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:43.217564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:43.217526    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972" event={"ID":"2fc4ea5a-d1e2-4007-91cc-53fb81272f97","Type":"ContainerDied","Data":"f27cf40e7e52fac6119bbc3e9f36b32c8f84de5d88a9fea3ce08550e35e6d49f"}
Oct 30 17:32:43.217790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:43.217627    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f27cf40e7e52fac6119bbc3e9f36b32c8f84de5d88a9fea3ce08550e35e6d49f"
Oct 30 17:32:43.217790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:43.217736    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"
Oct 30 17:32:43.391434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:43.391401    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:44.328078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:44.328042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:44.998780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:44.998745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:45.245089 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:32:45.245053    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:32:45.682980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.682946    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:45.795984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.795945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:45.942012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.941928    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"]
Oct 30 17:32:45.942627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.942608    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2fc4ea5a-d1e2-4007-91cc-53fb81272f97" containerName="revoke-keys"
Oct 30 17:32:45.942627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.942628    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc4ea5a-d1e2-4007-91cc-53fb81272f97" containerName="revoke-keys"
Oct 30 17:32:45.942791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.942687    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cd5859cd-3152-4a76-bd22-47ae1f877556" containerName="revoke-keys"
Oct 30 17:32:45.942791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.942693    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5859cd-3152-4a76-bd22-47ae1f877556" containerName="revoke-keys"
Oct 30 17:32:45.942853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.942810    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc4ea5a-d1e2-4007-91cc-53fb81272f97" containerName="revoke-keys"
Oct 30 17:32:45.942853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.942823    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5859cd-3152-4a76-bd22-47ae1f877556" containerName="revoke-keys"
Oct 30 17:32:45.946842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.946821    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:45.961709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.961630    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:45.973586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.973553    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:32:45.981886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:45.981853    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"]
Oct 30 17:32:46.071281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.071242    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.071787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.071314    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvf6b\" (UniqueName: \"kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.071787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.071470    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.172772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.172734    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.172947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.172808    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvf6b\" (UniqueName: \"kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.172947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.172879    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.173542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.173517    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.175388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.175359    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.209932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.209849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvf6b\" (UniqueName: \"kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b\") pod \"maas-api-revoke-sub-2600ca07-8hqs7\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.263445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.263408    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:46.583897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.583116    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"]
Oct 30 17:32:46.754855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:46.754805    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:47.043674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.040489    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"]
Oct 30 17:32:47.046791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.046251    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.188662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.187665    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"]
Oct 30 17:32:47.207663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.206779    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.207663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.207096    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7s9v\" (UniqueName: \"kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.207663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.207182    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.283171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.283067    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7" event={"ID":"80fa06d9-3f50-45fe-92c1-2d437c0fb543","Type":"ContainerStarted","Data":"564d65f49947ccdf76e3ce24fba84759fc3c18ac231b013072193cae9c05be1b"}
Oct 30 17:32:47.284726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.284695    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7" event={"ID":"80fa06d9-3f50-45fe-92c1-2d437c0fb543","Type":"ContainerStarted","Data":"4aafe26b3ee8e70be6c861d26b5d44dd51e94ef3cd92e777958e6ee7981eac79"}
Oct 30 17:32:47.312820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.312775    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.313341 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.313310    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7s9v\" (UniqueName: \"kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.313556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.313538    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.314573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.314550    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.324124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.324095    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.366709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.349561    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7s9v\" (UniqueName: \"kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v\") pod \"maas-api-revoke-sub-83dbc5c0-jmdfl\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:47.507732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:47.494913    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:48.985670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:48.960059    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:48.985670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:48.967821    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:32:48.985670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:48.968276    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.039686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.038781    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.084670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.081265    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:32:49.190670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.092188    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.190670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.165401    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.268719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.227971    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7" podStartSLOduration=4.227949612 podStartE2EDuration="4.227949612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:47.342269164 +0000 UTC m=+1173.104313001" watchObservedRunningTime="2026-10-30 17:32:49.227949612 +0000 UTC m=+1174.989993426"
Oct 30 17:32:49.272496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.272452    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"]
Oct 30 17:32:49.343167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.343120    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:32:49.345707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.345673    5534 generic.go:356] "Generic (PLEG): container finished" podID="80fa06d9-3f50-45fe-92c1-2d437c0fb543" containerID="564d65f49947ccdf76e3ce24fba84759fc3c18ac231b013072193cae9c05be1b" exitCode=0
Oct 30 17:32:49.366456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.345900    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7" event={"ID":"80fa06d9-3f50-45fe-92c1-2d437c0fb543","Type":"ContainerDied","Data":"564d65f49947ccdf76e3ce24fba84759fc3c18ac231b013072193cae9c05be1b"}
Oct 30 17:32:49.367804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.359786    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.857676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.854517    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.977924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.960229    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.977924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.961066    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:32:49.977924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.968851    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:32:49.977924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.969149    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:49.977924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.969408    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:32:49.977924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:49.969457    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:32:50.045687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.041577    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:32:50.045687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.041989    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:32:50.096026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.095988    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:50.096678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.096634    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:32:50.119008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.118962    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:32:50.119673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.119624    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:32:50.185668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.181974    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:50.217671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.193161    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:32:50.362694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.359001    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:32:50.362694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.359534    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:32:50.422695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.407802    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:50.750818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.742548    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:32:50.822334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.822296    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl" event={"ID":"bb630fd3-c4e7-463a-966e-8d99679bcef3","Type":"ContainerStarted","Data":"a8f0bc6abba7a8b2fcdf12dad3bd186a82ced10a29219c1e283b628ba9f5d9a8"}
Oct 30 17:32:50.858904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.858842    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:50.893950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.876758    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:32:50.965168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.965127    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:32:50.977665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.973260    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:50.977665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:50.973573    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:32:51.126682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.044268    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:51.319042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.196755    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:51.319042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.120734    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:51.319042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.117047    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:32:51.420766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.391489    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:32:51.475663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.421136    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:51.564675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.544372    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:51.684974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.655669    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:52.036732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:51.991127    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:32:52.049910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.023738    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:32:52.060670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.033838    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:32:52.061007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.037058    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:52.170318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.170265    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:32:52.197573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.197522    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:52.287961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.287850    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:32:52.319819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.319744    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:52.829866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:52.829813    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.076319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.073747    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.080809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.079772    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.091442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.091407    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj" podUID="5217b9b6-dc38-4b7e-bf14-0206e45811e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Oct 30 17:32:53.115751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.111204    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5kxv" podUID="ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.116928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.116897    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.177670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.118772    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.238989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.238932    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:32:53.300910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.299606    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.305323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.301111    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.434765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.417861    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:53.550667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.534291    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:53.584343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.584299    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:53.699733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.669910    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca\") pod \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") "
Oct 30 17:32:53.699733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.670043    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca\") pod \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") "
Oct 30 17:32:53.699733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.670084    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b\" (UniqueName: \"kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b\") pod \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\" (UID: \"80fa06d9-3f50-45fe-92c1-2d437c0fb543\") "
Oct 30 17:32:53.708226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.703813    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:32:53.728607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.728561    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca" pod "80fa06d9-3f50-45fe-92c1-2d437c0fb543" (UID: "80fa06d9-3f50-45fe-92c1-2d437c0fb543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:53.914294 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:32:53.909517    5534 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.471s"
Oct 30 17:32:53.922987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.920868    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:53.924797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.924757    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:32:53.936040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.936015    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:32:53.961713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.943076    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca" pod "80fa06d9-3f50-45fe-92c1-2d437c0fb543" (UID: "80fa06d9-3f50-45fe-92c1-2d437c0fb543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:53.969723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:53.964125    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b" pod "80fa06d9-3f50-45fe-92c1-2d437c0fb543" (UID: "80fa06d9-3f50-45fe-92c1-2d437c0fb543"). InnerVolumeSpecName "kube-api-access-pvf6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:54.032735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.032706    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:32:54.032975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.032962    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:32:54.033089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.033080    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:32:54.035101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.035082    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:32:54.035268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.035257    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:32:54.047662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.040358    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvf6b\" (UniqueName: \"kubernetes.io/projected/80fa06d9-3f50-45fe-92c1-2d437c0fb543-kube-api-access-pvf6b\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:54.047662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.040388    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fa06d9-3f50-45fe-92c1-2d437c0fb543-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:54.047662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.047095    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:54.054418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.051354    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:32:54.057313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.057293    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:32:54.075671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.074557    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:32:54.083071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.082063    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl" event={"ID":"bb630fd3-c4e7-463a-966e-8d99679bcef3","Type":"ContainerStarted","Data":"8759174a0a7fefa7129cd92c094bb25fd1c20331498cdd054c3d402241a872f8"}
Oct 30 17:32:54.093761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:54.093736    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:32:55.086942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.086906    5534 generic.go:356] "Generic (PLEG): container finished" podID="bb630fd3-c4e7-463a-966e-8d99679bcef3" containerID="8759174a0a7fefa7129cd92c094bb25fd1c20331498cdd054c3d402241a872f8" exitCode=0
Oct 30 17:32:55.087410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.087003    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl" event={"ID":"bb630fd3-c4e7-463a-966e-8d99679bcef3","Type":"ContainerDied","Data":"8759174a0a7fefa7129cd92c094bb25fd1c20331498cdd054c3d402241a872f8"}
Oct 30 17:32:55.088842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.088815    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7" event={"ID":"80fa06d9-3f50-45fe-92c1-2d437c0fb543","Type":"ContainerDied","Data":"4aafe26b3ee8e70be6c861d26b5d44dd51e94ef3cd92e777958e6ee7981eac79"}
Oct 30 17:32:55.088980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.088861    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aafe26b3ee8e70be6c861d26b5d44dd51e94ef3cd92e777958e6ee7981eac79"
Oct 30 17:32:55.088980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.088921    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"
Oct 30 17:32:55.255140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.255115    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:55.272046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.272021    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v\" (UniqueName: \"kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v\") pod \"bb630fd3-c4e7-463a-966e-8d99679bcef3\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") "
Oct 30 17:32:55.272203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.272115    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca\") pod \"bb630fd3-c4e7-463a-966e-8d99679bcef3\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") "
Oct 30 17:32:55.272203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.272180    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca\") pod \"bb630fd3-c4e7-463a-966e-8d99679bcef3\" (UID: \"bb630fd3-c4e7-463a-966e-8d99679bcef3\") "
Oct 30 17:32:55.272605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.272578    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca" pod "bb630fd3-c4e7-463a-966e-8d99679bcef3" (UID: "bb630fd3-c4e7-463a-966e-8d99679bcef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:55.272801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.272778    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:55.274398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.274366    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v" pod "bb630fd3-c4e7-463a-966e-8d99679bcef3" (UID: "bb630fd3-c4e7-463a-966e-8d99679bcef3"). InnerVolumeSpecName "kube-api-access-l7s9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:55.274963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.274936    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca" pod "bb630fd3-c4e7-463a-966e-8d99679bcef3" (UID: "bb630fd3-c4e7-463a-966e-8d99679bcef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:55.275932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.275901    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:55.373911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.373883    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb630fd3-c4e7-463a-966e-8d99679bcef3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:55.373911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:55.373908    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7s9v\" (UniqueName: \"kubernetes.io/projected/bb630fd3-c4e7-463a-966e-8d99679bcef3-kube-api-access-l7s9v\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:56.100377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.100344    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"
Oct 30 17:32:56.100929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.100471    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl" event={"ID":"bb630fd3-c4e7-463a-966e-8d99679bcef3","Type":"ContainerDied","Data":"a8f0bc6abba7a8b2fcdf12dad3bd186a82ced10a29219c1e283b628ba9f5d9a8"}
Oct 30 17:32:56.100929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.100529    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f0bc6abba7a8b2fcdf12dad3bd186a82ced10a29219c1e283b628ba9f5d9a8"
Oct 30 17:32:56.272384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.272348    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"]
Oct 30 17:32:56.273072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.273049    5534 cpu_manager.go:414] "containerMap: removing container" podUID="80fa06d9-3f50-45fe-92c1-2d437c0fb543" containerName="revoke-keys"
Oct 30 17:32:56.273072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.273071    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fa06d9-3f50-45fe-92c1-2d437c0fb543" containerName="revoke-keys"
Oct 30 17:32:56.273331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.273088    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bb630fd3-c4e7-463a-966e-8d99679bcef3" containerName="revoke-keys"
Oct 30 17:32:56.273331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.273095    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb630fd3-c4e7-463a-966e-8d99679bcef3" containerName="revoke-keys"
Oct 30 17:32:56.273331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.273288    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fa06d9-3f50-45fe-92c1-2d437c0fb543" containerName="revoke-keys"
Oct 30 17:32:56.273331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.273304    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb630fd3-c4e7-463a-966e-8d99679bcef3" containerName="revoke-keys"
Oct 30 17:32:56.277677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.277634    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.293358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.293329    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:56.295136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.295117    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"]
Oct 30 17:32:56.300794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.300773    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:32:56.387607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.387530    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rbz\" (UniqueName: \"kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.388060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.388036    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.388294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.388268    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.489447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.489414    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.489606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.489482    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.489606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.489544    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rbz\" (UniqueName: \"kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.490393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.490363    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.492061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.492036    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.512758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.512727    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rbz\" (UniqueName: \"kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz\") pod \"maas-api-revoke-sub-10e84683-vlhqf\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.593102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.593066    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:56.908248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.908201    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:56.911180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:56.911154    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"]
Oct 30 17:32:56.911847 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:32:56.911822    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3499c804_eb7e_4036_bec8_bbf10fd28082.slice/crio-88130075721f54892af92f0b9cba3d5ddc4bad337d1a99ea68f19d7c669265c8 WatchSource:0}: Error finding container 88130075721f54892af92f0b9cba3d5ddc4bad337d1a99ea68f19d7c669265c8: Status 404 returned error can't find the container with id 88130075721f54892af92f0b9cba3d5ddc4bad337d1a99ea68f19d7c669265c8
Oct 30 17:32:57.110310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:57.110275    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf" event={"ID":"3499c804-eb7e-4036-bec8-bbf10fd28082","Type":"ContainerStarted","Data":"88130075721f54892af92f0b9cba3d5ddc4bad337d1a99ea68f19d7c669265c8"}
Oct 30 17:32:58.121116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:58.121083    5534 generic.go:356] "Generic (PLEG): container finished" podID="3499c804-eb7e-4036-bec8-bbf10fd28082" containerID="62b62d4aadd8aab2b8dcb01838e3bbec8f4cf290d55329ea76eb3cb036e8948b" exitCode=0
Oct 30 17:32:58.121516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:58.121241    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf" event={"ID":"3499c804-eb7e-4036-bec8-bbf10fd28082","Type":"ContainerDied","Data":"62b62d4aadd8aab2b8dcb01838e3bbec8f4cf290d55329ea76eb3cb036e8948b"}
Oct 30 17:32:58.533349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:58.533315    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:58.647919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:58.647880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:32:59.304783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.304754    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:32:59.432731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.432627    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca\") pod \"3499c804-eb7e-4036-bec8-bbf10fd28082\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") "
Oct 30 17:32:59.432731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.432723    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca\") pod \"3499c804-eb7e-4036-bec8-bbf10fd28082\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") "
Oct 30 17:32:59.432970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.432843    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz\" (UniqueName: \"kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz\") pod \"3499c804-eb7e-4036-bec8-bbf10fd28082\" (UID: \"3499c804-eb7e-4036-bec8-bbf10fd28082\") "
Oct 30 17:32:59.433089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.433060    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca" pod "3499c804-eb7e-4036-bec8-bbf10fd28082" (UID: "3499c804-eb7e-4036-bec8-bbf10fd28082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:59.433657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.433625    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:59.435130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.435072    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca" pod "3499c804-eb7e-4036-bec8-bbf10fd28082" (UID: "3499c804-eb7e-4036-bec8-bbf10fd28082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:59.435259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.435156    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz" pod "3499c804-eb7e-4036-bec8-bbf10fd28082" (UID: "3499c804-eb7e-4036-bec8-bbf10fd28082"). InnerVolumeSpecName "kube-api-access-h9rbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:59.535044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.535007    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3499c804-eb7e-4036-bec8-bbf10fd28082-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:32:59.535044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:32:59.535042    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rbz\" (UniqueName: \"kubernetes.io/projected/3499c804-eb7e-4036-bec8-bbf10fd28082-kube-api-access-h9rbz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:00.138035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:00.138006    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"
Oct 30 17:33:00.138035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:00.138014    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf" event={"ID":"3499c804-eb7e-4036-bec8-bbf10fd28082","Type":"ContainerDied","Data":"88130075721f54892af92f0b9cba3d5ddc4bad337d1a99ea68f19d7c669265c8"}
Oct 30 17:33:00.138245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:00.138060    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88130075721f54892af92f0b9cba3d5ddc4bad337d1a99ea68f19d7c669265c8"
Oct 30 17:33:00.333748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:00.333705    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:00.939389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:00.939346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:01.063832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:01.063795    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:01.569477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:01.569444    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:02.035392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:02.035294    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:02.766267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:02.766228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:03.576767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.564348    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"]
Oct 30 17:33:03.576767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.565378    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3499c804-eb7e-4036-bec8-bbf10fd28082" containerName="revoke-keys"
Oct 30 17:33:03.576767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.565402    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3499c804-eb7e-4036-bec8-bbf10fd28082" containerName="revoke-keys"
Oct 30 17:33:03.576767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.565581    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3499c804-eb7e-4036-bec8-bbf10fd28082" containerName="revoke-keys"
Oct 30 17:33:03.591666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.583808    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:03.594108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.594083    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.626671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.615029    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f8ac7b58-openshift-default-dockercfg-w7bxb\""
Oct 30 17:33:03.644047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.644009    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"]
Oct 30 17:33:03.801814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.801777    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:03.827295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.827199    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.827674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.827624    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.827859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.827840    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.827983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.827965    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.828094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.828078    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frff\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.828216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.828202    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.828420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.828400    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.828575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.828558    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.842724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.842323    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.943855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.943819    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.944203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.944181    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.944333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.944318    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.944456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.944440    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.944564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.944551    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frff\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.947165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.947138    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.948079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.945471    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.948201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.947989    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.948434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.948409    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.948520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.948465    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.948520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.948494    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.949051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.949028    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.949234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.949211    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.949346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.949262    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.949413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.949282    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:03.952293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:03.952271    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:04.002669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.002506    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frff\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:04.016767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.013787    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token\") pod \"e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") " pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:04.143097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.143059    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:04.307262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.307226    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:04.648376 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:04.648345    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac9d291_b3ce_48a4_bf07_8b36709a01ff.slice/crio-a94518c27eecf860e468df3e2b047fd370e3c7a0802d363d8df41f4d3739411d WatchSource:0}: Error finding container a94518c27eecf860e468df3e2b047fd370e3c7a0802d363d8df41f4d3739411d: Status 404 returned error can't find the container with id a94518c27eecf860e468df3e2b047fd370e3c7a0802d363d8df41f4d3739411d
Oct 30 17:33:04.649705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.649677    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"]
Oct 30 17:33:04.651077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.651040    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:04.651155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.651112    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:04.651155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:04.651142    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:05.614864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:05.614828    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" event={"ID":"bac9d291-b3ce-48a4-bf07-8b36709a01ff","Type":"ContainerStarted","Data":"87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"}
Oct 30 17:33:05.615436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:05.614888    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" event={"ID":"bac9d291-b3ce-48a4-bf07-8b36709a01ff","Type":"ContainerStarted","Data":"a94518c27eecf860e468df3e2b047fd370e3c7a0802d363d8df41f4d3739411d"}
Oct 30 17:33:05.677521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:05.677456    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podStartSLOduration=2.67743629 podStartE2EDuration="2.67743629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:05.672205356 +0000 UTC m=+1191.434249187" watchObservedRunningTime="2026-10-30 17:33:05.67743629 +0000 UTC m=+1191.439480104"
Oct 30 17:33:06.073115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.073018    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:06.160914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.160884    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"]
Oct 30 17:33:06.165489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.165462    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.187313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.187278    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"]
Oct 30 17:33:06.192075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.192043    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:06.192211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.192150    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:33:06.282089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.282060    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.282290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.282186    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4njc\" (UniqueName: \"kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.282374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.282291    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.308424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.308377    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:06.315051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.315020    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:06.384169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.384131    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.384357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.384197    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4njc\" (UniqueName: \"kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.384357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.384295    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.385007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.384983    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.386917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.386899    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.407923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.407854    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4njc\" (UniqueName: \"kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc\") pod \"maas-api-revoke-sub-5096dbee-v8942\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.491185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.491144    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:06.515182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.515151    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:06.619712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.619671    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"]
Oct 30 17:33:06.631407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.631361    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.638794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.636671    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:06.638794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.638228    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:33:06.650396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.650348    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 17:33:06.650630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.650457    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 17:33:06.650630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.650523    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 17:33:06.655971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.655853    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"]
Oct 30 17:33:06.668768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.668738    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m72mh\""
Oct 30 17:33:06.790766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.790725    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.790967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.790890    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.790967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.790941    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.791050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.791015    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.791097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.791063    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.791097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.791094    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.791166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.791115    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csf55\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.817057 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:06.817025    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aabf6f5_b57e_4168_a888_193f241790b3.slice/crio-d27e36252190c0ccb054ff1e250a90b00c4dca0a300e974faf188afe602e3a93 WatchSource:0}: Error finding container d27e36252190c0ccb054ff1e250a90b00c4dca0a300e974faf188afe602e3a93: Status 404 returned error can't find the container with id d27e36252190c0ccb054ff1e250a90b00c4dca0a300e974faf188afe602e3a93
Oct 30 17:33:06.820556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.820530    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"]
Oct 30 17:33:06.892747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.892634    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.892747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.892720    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.892986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.892758    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.892986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.892785    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csf55\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.892986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.892891    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.893157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.893037    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.893157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.893106    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.894317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.894281    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.895788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.895763    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.895897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.895810    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.895897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.895864    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.896044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.896020    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.915846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.915822    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.917089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.917067    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csf55\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t66m4\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:06.976571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:06.976534    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:07.327478 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:07.327442    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e977fe_38c6_4e11_8ada_3f55a8023fd3.slice/crio-1c7fa622add4d6ee1f64f0435aeb5d8cccd95fd07f9e401c63cd912437545335 WatchSource:0}: Error finding container 1c7fa622add4d6ee1f64f0435aeb5d8cccd95fd07f9e401c63cd912437545335: Status 404 returned error can't find the container with id 1c7fa622add4d6ee1f64f0435aeb5d8cccd95fd07f9e401c63cd912437545335
Oct 30 17:33:07.327783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.327764    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"]
Oct 30 17:33:07.334063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.334030    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:07.444020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.443977    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:07.642589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.642552    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" event={"ID":"20e977fe-38c6-4e11-8ada-3f55a8023fd3","Type":"ContainerStarted","Data":"1c7fa622add4d6ee1f64f0435aeb5d8cccd95fd07f9e401c63cd912437545335"}
Oct 30 17:33:07.644673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.644619    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942" event={"ID":"8aabf6f5-b57e-4168-a888-193f241790b3","Type":"ContainerStarted","Data":"59093c98acfb2bb83aa04ea5d747e608aea4e09c97e91850d76a7fcfad2612fa"}
Oct 30 17:33:07.644848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.644692    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942" event={"ID":"8aabf6f5-b57e-4168-a888-193f241790b3","Type":"ContainerStarted","Data":"d27e36252190c0ccb054ff1e250a90b00c4dca0a300e974faf188afe602e3a93"}
Oct 30 17:33:07.688084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.688013    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942" podStartSLOduration=1.687995219 podStartE2EDuration="1.687995219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:07.687355482 +0000 UTC m=+1193.449399296" watchObservedRunningTime="2026-10-30 17:33:07.687995219 +0000 UTC m=+1193.450039042"
Oct 30 17:33:07.827661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:07.827614    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:08.660854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.659094    5534 generic.go:356] "Generic (PLEG): container finished" podID="8aabf6f5-b57e-4168-a888-193f241790b3" containerID="59093c98acfb2bb83aa04ea5d747e608aea4e09c97e91850d76a7fcfad2612fa" exitCode=0
Oct 30 17:33:08.660854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.660768    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942" event={"ID":"8aabf6f5-b57e-4168-a888-193f241790b3","Type":"ContainerDied","Data":"59093c98acfb2bb83aa04ea5d747e608aea4e09c97e91850d76a7fcfad2612fa"}
Oct 30 17:33:08.828898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.828852    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"]
Oct 30 17:33:08.836774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.835817    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.875438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.875402    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"]
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.925980    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926053    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926089    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926131    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926157    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtbrq\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926201    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926224    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926285    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:08.926688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:08.926326    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.027852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.027812    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.027912    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.027957    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028013    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028048    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtbrq\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028088    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028116    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028172    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028205    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.028603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.028586    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.029122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.029105    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.029310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.029293    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.031456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.031412    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.031597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.031571    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.033055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.033028    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.034188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.034145    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.057333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.056275    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.060678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.058107    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtbrq\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.166424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.166387    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:09.514466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.514437    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"]
Oct 30 17:33:09.516188 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:09.516155    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85be8266_684d_42b7_9940_a9d096dedaa4.slice/crio-005f8cacbb5df4af6ddfdfd7f6dbe6be10d03fb0439512608a3f12ff928ea00e WatchSource:0}: Error finding container 005f8cacbb5df4af6ddfdfd7f6dbe6be10d03fb0439512608a3f12ff928ea00e: Status 404 returned error can't find the container with id 005f8cacbb5df4af6ddfdfd7f6dbe6be10d03fb0439512608a3f12ff928ea00e
Oct 30 17:33:09.519354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.519317    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:09.519459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.519409    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:09.519570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.519465    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:09.687418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.687314    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" event={"ID":"85be8266-684d-42b7-9940-a9d096dedaa4","Type":"ContainerStarted","Data":"703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be"}
Oct 30 17:33:09.687418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.687373    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" event={"ID":"85be8266-684d-42b7-9940-a9d096dedaa4","Type":"ContainerStarted","Data":"005f8cacbb5df4af6ddfdfd7f6dbe6be10d03fb0439512608a3f12ff928ea00e"}
Oct 30 17:33:09.786991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.786867    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podStartSLOduration=1.786827153 podStartE2EDuration="1.786827153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:09.753019144 +0000 UTC m=+1195.515062960" watchObservedRunningTime="2026-10-30 17:33:09.786827153 +0000 UTC m=+1195.548870962"
Oct 30 17:33:09.794133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.794097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:09.933943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:09.933370    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:10.035407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.034944    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:10.115330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.113903    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:10.164671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.139358    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca\") pod \"8aabf6f5-b57e-4168-a888-193f241790b3\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") "
Oct 30 17:33:10.164671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.139419    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc\" (UniqueName: \"kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc\") pod \"8aabf6f5-b57e-4168-a888-193f241790b3\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") "
Oct 30 17:33:10.164671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.139689    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca\") pod \"8aabf6f5-b57e-4168-a888-193f241790b3\" (UID: \"8aabf6f5-b57e-4168-a888-193f241790b3\") "
Oct 30 17:33:10.206822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.182975    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:10.206822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.197474    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:10.206822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.197530    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:10.226837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.226775    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca" pod "8aabf6f5-b57e-4168-a888-193f241790b3" (UID: "8aabf6f5-b57e-4168-a888-193f241790b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:10.235100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.235035    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca" pod "8aabf6f5-b57e-4168-a888-193f241790b3" (UID: "8aabf6f5-b57e-4168-a888-193f241790b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:10.239411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.239278    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:10.249965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.249922    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc" pod "8aabf6f5-b57e-4168-a888-193f241790b3" (UID: "8aabf6f5-b57e-4168-a888-193f241790b3"). InnerVolumeSpecName "kube-api-access-b4njc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:10.264666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.259850    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.264666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.259897    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aabf6f5-b57e-4168-a888-193f241790b3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.264666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.259911    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4njc\" (UniqueName: \"kubernetes.io/projected/8aabf6f5-b57e-4168-a888-193f241790b3-kube-api-access-b4njc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.460322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.460171    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:10.700594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.700132    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"
Oct 30 17:33:10.701005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.700781    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942" event={"ID":"8aabf6f5-b57e-4168-a888-193f241790b3","Type":"ContainerDied","Data":"d27e36252190c0ccb054ff1e250a90b00c4dca0a300e974faf188afe602e3a93"}
Oct 30 17:33:10.701005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.700828    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27e36252190c0ccb054ff1e250a90b00c4dca0a300e974faf188afe602e3a93"
Oct 30 17:33:10.945668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:10.941503    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:11.234347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.233803    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:11.234347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.233860    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:11.394760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.394705    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"]
Oct 30 17:33:11.396076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.396041    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8aabf6f5-b57e-4168-a888-193f241790b3" containerName="revoke-keys"
Oct 30 17:33:11.396145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.396080    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aabf6f5-b57e-4168-a888-193f241790b3" containerName="revoke-keys"
Oct 30 17:33:11.396251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.396239    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aabf6f5-b57e-4168-a888-193f241790b3" containerName="revoke-keys"
Oct 30 17:33:11.403973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.403952    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.415832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.415799    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"]
Oct 30 17:33:11.422170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.422148    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:33:11.422847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.422820    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:11.523226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.523151    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:11.529947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.529912    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8nv\" (UniqueName: \"kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.530073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.530016    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.530112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.530091    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.632155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.631717    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8nv\" (UniqueName: \"kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.632155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.631776    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.632155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.631837    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.632926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.632785    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.676139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.676071    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.699879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.699785    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8nv\" (UniqueName: \"kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv\") pod \"maas-api-revoke-sub-8453eb74-svgmp\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:11.731978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:11.731935    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:12.074284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.074197    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:12.138086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.138038    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"]
Oct 30 17:33:12.196476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.196430    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:12.196689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.196495    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:12.671716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.652865    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:12.671716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.652947    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:33:12.729495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.729437    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp" event={"ID":"24accd9e-69f3-47ef-a8d7-2cb1207d6224","Type":"ContainerStarted","Data":"e9546421708917ec67ff3a7104099e30dae0bc735ccc34f73cf73a0f60e091a1"}
Oct 30 17:33:12.729669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.729577    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp" event={"ID":"24accd9e-69f3-47ef-a8d7-2cb1207d6224","Type":"ContainerStarted","Data":"e43f38d42f77249442ff3550bcc032020162a04b0ec3006c387c6598d37a8510"}
Oct 30 17:33:12.794669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:12.794538    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp" podStartSLOduration=1.794519643 podStartE2EDuration="1.794519643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:12.769243884 +0000 UTC m=+1198.531287698" watchObservedRunningTime="2026-10-30 17:33:12.794519643 +0000 UTC m=+1198.556563460"
Oct 30 17:33:13.174671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.174054    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:13.174671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.174113    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:13.745529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.744696    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" event={"ID":"20e977fe-38c6-4e11-8ada-3f55a8023fd3","Type":"ContainerStarted","Data":"088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17"}
Oct 30 17:33:13.745529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.745124    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:13.748337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.746957    5534 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t66m4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:33:13.748337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.747006    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" podUID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:33:13.748337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.747374    5534 generic.go:356] "Generic (PLEG): container finished" podID="24accd9e-69f3-47ef-a8d7-2cb1207d6224" containerID="e9546421708917ec67ff3a7104099e30dae0bc735ccc34f73cf73a0f60e091a1" exitCode=0
Oct 30 17:33:13.748337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.747408    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp" event={"ID":"24accd9e-69f3-47ef-a8d7-2cb1207d6224","Type":"ContainerDied","Data":"e9546421708917ec67ff3a7104099e30dae0bc735ccc34f73cf73a0f60e091a1"}
Oct 30 17:33:13.808667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:13.807590    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" podStartSLOduration=2.485516342 podStartE2EDuration="7.80756866s" totalImagesPullingTime="5.322052318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:13.80408909 +0000 UTC m=+1199.566132905" watchObservedRunningTime="2026-10-30 17:33:13.80756866 +0000 UTC m=+1199.569612519"
Oct 30 17:33:14.111674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:14.111501    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:14.188668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:14.172061    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:14.188668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:14.172136    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:14.797673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:14.792054    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:33:15.045997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.045971    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:15.143455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.142716    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca\") pod \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") "
Oct 30 17:33:15.143455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.142807    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca\") pod \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") "
Oct 30 17:33:15.143455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.142917    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv\" (UniqueName: \"kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv\") pod \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\" (UID: \"24accd9e-69f3-47ef-a8d7-2cb1207d6224\") "
Oct 30 17:33:15.143883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.143857    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca" pod "24accd9e-69f3-47ef-a8d7-2cb1207d6224" (UID: "24accd9e-69f3-47ef-a8d7-2cb1207d6224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:15.145939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.145911    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv" pod "24accd9e-69f3-47ef-a8d7-2cb1207d6224" (UID: "24accd9e-69f3-47ef-a8d7-2cb1207d6224"). InnerVolumeSpecName "kube-api-access-jh8nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:15.146052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.146001    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca" pod "24accd9e-69f3-47ef-a8d7-2cb1207d6224" (UID: "24accd9e-69f3-47ef-a8d7-2cb1207d6224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:15.183148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.173001    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:15.183148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.173150    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:15.229764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.229728    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:15.245111 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:33:15.245058    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:33:15.245246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.245183    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:15.245246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.245209    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24accd9e-69f3-47ef-a8d7-2cb1207d6224-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:15.245246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.245220    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8nv\" (UniqueName: \"kubernetes.io/projected/24accd9e-69f3-47ef-a8d7-2cb1207d6224-kube-api-access-jh8nv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:15.336699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.336586    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:15.530945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.530916    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:15.790996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.790966    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"
Oct 30 17:33:15.790996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.790975    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp" event={"ID":"24accd9e-69f3-47ef-a8d7-2cb1207d6224","Type":"ContainerDied","Data":"e43f38d42f77249442ff3550bcc032020162a04b0ec3006c387c6598d37a8510"}
Oct 30 17:33:15.791203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:15.791030    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e43f38d42f77249442ff3550bcc032020162a04b0ec3006c387c6598d37a8510"
Oct 30 17:33:16.085722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:16.081473    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:16.172669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:16.172160    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:16.172669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:16.172234    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:16.298731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:16.296740    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:16.456673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:16.452963    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:17.044608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:17.044574    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:17.168346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:17.168314    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 17:33:17.168900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:17.168385    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 17:33:18.173570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:18.173537    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:18.175178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:18.175156    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:18.176157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:18.176135    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:33:18.402466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:18.402431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:19.604790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:19.604756    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:19.723167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:19.723133    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:20.553253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:20.553199    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:21.075698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:21.075669    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"]
Oct 30 17:33:21.136717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:21.132633    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46935f91-kh28k"]
Oct 30 17:33:21.987512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:21.987470    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:22.028781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.028736    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:22.053290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.053227    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:22.065769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.065717    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:22.086669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.082729    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:33:22.123021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.122981    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:22.217670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.198769    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:33:22.263391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.263274    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:22.324899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.324853    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:22.325535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.325503    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:33:22.463270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.463223    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:22.470680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.468404    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:23.007664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:22.930176    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 30 17:33:23.032668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.004970    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:23.104707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.085068    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.108747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.108086    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:23.183668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.183385    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.208668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.202582    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.208668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.203727    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.221672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.220823    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648c76ad-7072-49c4-95ee-3ff8c79738dc" path="/var/lib/kubelet/pods/648c76ad-7072-49c4-95ee-3ff8c79738dc/volumes"
Oct 30 17:33:23.221862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.221677    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"]
Oct 30 17:33:23.227674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.223594    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.250701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.248707    5534 cpu_manager.go:414] "containerMap: removing container" podUID="24accd9e-69f3-47ef-a8d7-2cb1207d6224" containerName="revoke-keys"
Oct 30 17:33:23.250701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.248734    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24accd9e-69f3-47ef-a8d7-2cb1207d6224" containerName="revoke-keys"
Oct 30 17:33:23.250701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.249104    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="24accd9e-69f3-47ef-a8d7-2cb1207d6224" containerName="revoke-keys"
Oct 30 17:33:23.339011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.338893    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.340976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.340947    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:33:23.435532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.435497    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"]
Oct 30 17:33:23.445566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.445540    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:33:23.450750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.450726    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:33:23.473839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.473807    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:33:23.474134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.474116    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:33:23.474312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.474298    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:33:23.474509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.474494    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:33:23.478680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.474681    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:33:23.478903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.478888    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:33:23.479065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.479054    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:33:23.479240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.479209    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:23.491572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.448287    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.504766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.502132    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:33:23.553020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.552988    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:23.665020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.664984    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:33:23.685672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.685335    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:33:23.735703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.724914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq66\" (UniqueName: \"kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.735703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.724971    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.735703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.725293    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.835675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.834921    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.835675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.835088    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq66\" (UniqueName: \"kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.835675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.835144    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:23.913100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.869910    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:24.006698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.990923    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:24.006698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:23.991735    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 17:33:24.056669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.040833    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:24.170725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.151966    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:33:24.170725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.155356    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:24.203679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.201524    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:24.207771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.207719    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:24.214844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.214023    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:24.224790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.224751    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:24.227307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.227215    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:33:24.290661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.290598    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:33:24.326336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.326305    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq66\" (UniqueName: \"kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66\") pod \"maas-api-revoke-sub-1417ef52-d58dl\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:24.347134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.344340    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:24.347134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.344611    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:24.481231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.481145    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:33:24.481231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.481204    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:33:24.513565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.513528    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:33:24.530973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.530948    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:24.536357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.535616    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:33:24.536357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.535986    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:33:24.542020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.540448    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:33:24.950038 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:24.950006    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cad0190_8798_4785_972f_6801410c9ba9.slice/crio-4696b58427ad1a638b494bbf6e0d07acd45c7ce7ebe28f77439e5203b7f2c170 WatchSource:0}: Error finding container 4696b58427ad1a638b494bbf6e0d07acd45c7ce7ebe28f77439e5203b7f2c170: Status 404 returned error can't find the container with id 4696b58427ad1a638b494bbf6e0d07acd45c7ce7ebe28f77439e5203b7f2c170
Oct 30 17:33:24.957193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:24.957167    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"]
Oct 30 17:33:25.764955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:25.764857    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl" event={"ID":"5cad0190-8798-4785-972f-6801410c9ba9","Type":"ContainerStarted","Data":"f37c0228dbb26b0cb273bef00995341e8335f7f8fe4694eb3ea968d81f0429f4"}
Oct 30 17:33:25.765359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:25.764976    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl" event={"ID":"5cad0190-8798-4785-972f-6801410c9ba9","Type":"ContainerStarted","Data":"4696b58427ad1a638b494bbf6e0d07acd45c7ce7ebe28f77439e5203b7f2c170"}
Oct 30 17:33:25.809264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:25.809207    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl" podStartSLOduration=3.809189035 podStartE2EDuration="3.809189035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:25.80487454 +0000 UTC m=+1211.566918380" watchObservedRunningTime="2026-10-30 17:33:25.809189035 +0000 UTC m=+1211.571232849"
Oct 30 17:33:25.980827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:25.980793    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:26.092611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.092578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:26.205747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.205702    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:26.609277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.609249    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"]
Oct 30 17:33:26.613938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.613921    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.634063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.634036    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"]
Oct 30 17:33:26.705399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.705363    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.705594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.705517    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.705594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.705581    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xmj\" (UniqueName: \"kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.787318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.787280    5534 generic.go:356] "Generic (PLEG): container finished" podID="5cad0190-8798-4785-972f-6801410c9ba9" containerID="f37c0228dbb26b0cb273bef00995341e8335f7f8fe4694eb3ea968d81f0429f4" exitCode=0
Oct 30 17:33:26.787889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.787567    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl" event={"ID":"5cad0190-8798-4785-972f-6801410c9ba9","Type":"ContainerDied","Data":"f37c0228dbb26b0cb273bef00995341e8335f7f8fe4694eb3ea968d81f0429f4"}
Oct 30 17:33:26.807719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.807686    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.807913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.807810    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.807913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.807850    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xmj\" (UniqueName: \"kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.808474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.808449    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.810453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.810431    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.831684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.831628    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xmj\" (UniqueName: \"kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj\") pod \"maas-api-revoke-sub-5714b693-6vsb4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:26.927856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:26.927766    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:27.247511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:27.246936    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"]
Oct 30 17:33:27.256562 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:27.256527    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56963b27_c254_4064_8dd5_519fff13e3c4.slice/crio-6017ae7588bf3e90477026c9106530d5e9afbb29a4a57108c8c09e208b1e74a3 WatchSource:0}: Error finding container 6017ae7588bf3e90477026c9106530d5e9afbb29a4a57108c8c09e208b1e74a3: Status 404 returned error can't find the container with id 6017ae7588bf3e90477026c9106530d5e9afbb29a4a57108c8c09e208b1e74a3
Oct 30 17:33:27.799105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:27.799063    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4" event={"ID":"56963b27-c254-4064-8dd5-519fff13e3c4","Type":"ContainerStarted","Data":"066136b9d46fb306872d0d958a23c40ddaec73567cf4500896263c4e9a71fed7"}
Oct 30 17:33:27.799629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:27.799155    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4" event={"ID":"56963b27-c254-4064-8dd5-519fff13e3c4","Type":"ContainerStarted","Data":"6017ae7588bf3e90477026c9106530d5e9afbb29a4a57108c8c09e208b1e74a3"}
Oct 30 17:33:27.799826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:27.799683    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:27.848330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:27.848224    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4" podStartSLOduration=1.848199648 podStartE2EDuration="1.848199648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:27.8414981 +0000 UTC m=+1213.603541917" watchObservedRunningTime="2026-10-30 17:33:27.848199648 +0000 UTC m=+1213.610243461"
Oct 30 17:33:28.023913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.023889    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:28.128968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.128936    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca\") pod \"5cad0190-8798-4785-972f-6801410c9ba9\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") "
Oct 30 17:33:28.129165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.129037    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca\") pod \"5cad0190-8798-4785-972f-6801410c9ba9\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") "
Oct 30 17:33:28.129165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.129083    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66\" (UniqueName: \"kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66\") pod \"5cad0190-8798-4785-972f-6801410c9ba9\" (UID: \"5cad0190-8798-4785-972f-6801410c9ba9\") "
Oct 30 17:33:28.129325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.129288    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca" pod "5cad0190-8798-4785-972f-6801410c9ba9" (UID: "5cad0190-8798-4785-972f-6801410c9ba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:28.129702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.129684    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:28.131305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.131281    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca" pod "5cad0190-8798-4785-972f-6801410c9ba9" (UID: "5cad0190-8798-4785-972f-6801410c9ba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:28.131395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.131324    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66" pod "5cad0190-8798-4785-972f-6801410c9ba9" (UID: "5cad0190-8798-4785-972f-6801410c9ba9"). InnerVolumeSpecName "kube-api-access-nnq66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:28.231508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.231467    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cad0190-8798-4785-972f-6801410c9ba9-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:28.231508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.231507    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnq66\" (UniqueName: \"kubernetes.io/projected/5cad0190-8798-4785-972f-6801410c9ba9-kube-api-access-nnq66\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:28.809104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.809014    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"
Oct 30 17:33:28.809104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.809052    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl" event={"ID":"5cad0190-8798-4785-972f-6801410c9ba9","Type":"ContainerDied","Data":"4696b58427ad1a638b494bbf6e0d07acd45c7ce7ebe28f77439e5203b7f2c170"}
Oct 30 17:33:28.809104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.809098    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4696b58427ad1a638b494bbf6e0d07acd45c7ce7ebe28f77439e5203b7f2c170"
Oct 30 17:33:28.811171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.811126    5534 generic.go:356] "Generic (PLEG): container finished" podID="56963b27-c254-4064-8dd5-519fff13e3c4" containerID="066136b9d46fb306872d0d958a23c40ddaec73567cf4500896263c4e9a71fed7" exitCode=0
Oct 30 17:33:28.811295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:28.811208    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4" event={"ID":"56963b27-c254-4064-8dd5-519fff13e3c4","Type":"ContainerDied","Data":"066136b9d46fb306872d0d958a23c40ddaec73567cf4500896263c4e9a71fed7"}
Oct 30 17:33:29.367679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:29.367616    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:29.762741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:29.762613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:30.027377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.027295    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:30.164354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.164308    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca\") pod \"56963b27-c254-4064-8dd5-519fff13e3c4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") "
Oct 30 17:33:30.164576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.164446    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca\") pod \"56963b27-c254-4064-8dd5-519fff13e3c4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") "
Oct 30 17:33:30.164576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.164477    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj\" (UniqueName: \"kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj\") pod \"56963b27-c254-4064-8dd5-519fff13e3c4\" (UID: \"56963b27-c254-4064-8dd5-519fff13e3c4\") "
Oct 30 17:33:30.164859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.164829    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca" pod "56963b27-c254-4064-8dd5-519fff13e3c4" (UID: "56963b27-c254-4064-8dd5-519fff13e3c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:30.165148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.165132    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:30.166750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.166726    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca" pod "56963b27-c254-4064-8dd5-519fff13e3c4" (UID: "56963b27-c254-4064-8dd5-519fff13e3c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:30.166853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.166771    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj" pod "56963b27-c254-4064-8dd5-519fff13e3c4" (UID: "56963b27-c254-4064-8dd5-519fff13e3c4"). InnerVolumeSpecName "kube-api-access-w2xmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:30.266859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.266825    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xmj\" (UniqueName: \"kubernetes.io/projected/56963b27-c254-4064-8dd5-519fff13e3c4-kube-api-access-w2xmj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:30.266859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.266855    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56963b27-c254-4064-8dd5-519fff13e3c4-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:30.843475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.843443    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"
Oct 30 17:33:30.843675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.843483    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4" event={"ID":"56963b27-c254-4064-8dd5-519fff13e3c4","Type":"ContainerDied","Data":"6017ae7588bf3e90477026c9106530d5e9afbb29a4a57108c8c09e208b1e74a3"}
Oct 30 17:33:30.843675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:30.843530    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6017ae7588bf3e90477026c9106530d5e9afbb29a4a57108c8c09e208b1e74a3"
Oct 30 17:33:31.410854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:31.410817    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:32.961791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.961758    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"]
Oct 30 17:33:32.962620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.962601    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5cad0190-8798-4785-972f-6801410c9ba9" containerName="revoke-keys"
Oct 30 17:33:32.962718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.962622    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cad0190-8798-4785-972f-6801410c9ba9" containerName="revoke-keys"
Oct 30 17:33:32.962718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.962687    5534 cpu_manager.go:414] "containerMap: removing container" podUID="56963b27-c254-4064-8dd5-519fff13e3c4" containerName="revoke-keys"
Oct 30 17:33:32.962718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.962695    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56963b27-c254-4064-8dd5-519fff13e3c4" containerName="revoke-keys"
Oct 30 17:33:32.962839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.962828    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="56963b27-c254-4064-8dd5-519fff13e3c4" containerName="revoke-keys"
Oct 30 17:33:32.962875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.962849    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cad0190-8798-4785-972f-6801410c9ba9" containerName="revoke-keys"
Oct 30 17:33:32.966233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.966218    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:32.981275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:32.981244    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-84af0117-serving-cert\""
Oct 30 17:33:33.000271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.000234    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"]
Oct 30 17:33:33.057322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.057272    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:33.111107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.111073    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.111307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.111197    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgwl\" (UniqueName: \"kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.111307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.111276    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.111474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.111452    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.184632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.184592    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:33.212975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.212848    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.214593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.213892    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.214832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.213635    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.215482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.215070    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgwl\" (UniqueName: \"kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.215621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.215579    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.219973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.219948    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.220377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.220357    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.239232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.239201    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgwl\" (UniqueName: \"kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl\") pod \"maas-api-e2e-migrate-84af0117-79d8c75d49-764tt\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.279917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.279869    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:33.298289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.298220    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:33.598346 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:33.598298    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b62c09c_e2ad_4830_b9c0_0b7cc831e31a.slice/crio-c79f5f545eb309a7eae2c1b44b246e7c5e63f53ad7f77b803ad8873d11cd5dc8 WatchSource:0}: Error finding container c79f5f545eb309a7eae2c1b44b246e7c5e63f53ad7f77b803ad8873d11cd5dc8: Status 404 returned error can't find the container with id c79f5f545eb309a7eae2c1b44b246e7c5e63f53ad7f77b803ad8873d11cd5dc8
Oct 30 17:33:33.600586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.600557    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:33:33.600586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.600579    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"]
Oct 30 17:33:33.875951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.875857    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt" event={"ID":"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a","Type":"ContainerStarted","Data":"c79f5f545eb309a7eae2c1b44b246e7c5e63f53ad7f77b803ad8873d11cd5dc8"}
Oct 30 17:33:33.908841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:33.908808    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:34.027685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:34.027628    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:34.173306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:34.173201    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:34.888353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:34.888309    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt" event={"ID":"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a","Type":"ContainerStarted","Data":"426753c768c6773b25bbec36c0e06bea3dfa54659e33b688a0845388ebeb59ea"}
Oct 30 17:33:34.888546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:34.888427    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:34.935777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:34.935717    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt" podStartSLOduration=2.5798808429999998 podStartE2EDuration="2.935698367s" totalImagesPullingTime="355.817524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:34.930888306 +0000 UTC m=+1220.692932119" watchObservedRunningTime="2026-10-30 17:33:34.935698367 +0000 UTC m=+1220.697742183"
Oct 30 17:33:34.959757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:34.959724    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:35.957739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:35.957707    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"]
Oct 30 17:33:35.963290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:35.963269    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:35.979682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:35.979657    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"]
Oct 30 17:33:35.985130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:35.985108    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:35.985242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:35.985214    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:33:36.070949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.070918    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.071114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.071065    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2r4r\" (UniqueName: \"kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.071234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.071212    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.172112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.172085    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.172293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.172192    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2r4r\" (UniqueName: \"kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.172363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.172345    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.172794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.172775    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.174966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.174943    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.195093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.195070    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2r4r\" (UniqueName: \"kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r\") pod \"maas-api-revoke-sub-0c588e8b-pwqlc\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.280676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.280573    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:36.589710 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:33:36.589676    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3127ff94_efc3_4699_bfdf_0f38fcccf2fd.slice/crio-62fe5060035496abe8e16f46641f1770147fa90c2bdcc0f3949f09a7f84677ae WatchSource:0}: Error finding container 62fe5060035496abe8e16f46641f1770147fa90c2bdcc0f3949f09a7f84677ae: Status 404 returned error can't find the container with id 62fe5060035496abe8e16f46641f1770147fa90c2bdcc0f3949f09a7f84677ae
Oct 30 17:33:36.591774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.591749    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"]
Oct 30 17:33:36.700982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.700946    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:36.907753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:36.907713    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc" event={"ID":"3127ff94-efc3-4699-bfdf-0f38fcccf2fd","Type":"ContainerStarted","Data":"62fe5060035496abe8e16f46641f1770147fa90c2bdcc0f3949f09a7f84677ae"}
Oct 30 17:33:37.034607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:37.034563    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"]
Oct 30 17:33:37.039308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:37.039281    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9811f2-b7q2t"]
Oct 30 17:33:37.918349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:37.918311    5534 generic.go:356] "Generic (PLEG): container finished" podID="3127ff94-efc3-4699-bfdf-0f38fcccf2fd" containerID="01cd637a0b60bd0829b5f1dce4f5c5638e360e292ee5e40e76bccd41a6e62d64" exitCode=0
Oct 30 17:33:37.918540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:37.918396    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc" event={"ID":"3127ff94-efc3-4699-bfdf-0f38fcccf2fd","Type":"ContainerDied","Data":"01cd637a0b60bd0829b5f1dce4f5c5638e360e292ee5e40e76bccd41a6e62d64"}
Oct 30 17:33:38.371544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:38.371509    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da9ebc5-7bef-4170-a28d-f948e2eba7a2" path="/var/lib/kubelet/pods/6da9ebc5-7bef-4170-a28d-f948e2eba7a2/volumes"
Oct 30 17:33:38.498270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:38.498236    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:38.609753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:38.609715    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:39.134105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.134081    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:39.230312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.230220    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r\" (UniqueName: \"kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r\") pod \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") "
Oct 30 17:33:39.230485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.230273    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca\") pod \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") "
Oct 30 17:33:39.230485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.230475    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca\") pod \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\" (UID: \"3127ff94-efc3-4699-bfdf-0f38fcccf2fd\") "
Oct 30 17:33:39.233010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.232982    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca" pod "3127ff94-efc3-4699-bfdf-0f38fcccf2fd" (UID: "3127ff94-efc3-4699-bfdf-0f38fcccf2fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:39.234867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.234009    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca" pod "3127ff94-efc3-4699-bfdf-0f38fcccf2fd" (UID: "3127ff94-efc3-4699-bfdf-0f38fcccf2fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:39.239682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.237133    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r" pod "3127ff94-efc3-4699-bfdf-0f38fcccf2fd" (UID: "3127ff94-efc3-4699-bfdf-0f38fcccf2fd"). InnerVolumeSpecName "kube-api-access-k2r4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:39.332923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.332889    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:39.332923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.332917    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2r4r\" (UniqueName: \"kubernetes.io/projected/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-kube-api-access-k2r4r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:39.332923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.332927    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3127ff94-efc3-4699-bfdf-0f38fcccf2fd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:39.402876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.402835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:39.937059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.937032    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"
Oct 30 17:33:39.937254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.937064    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc" event={"ID":"3127ff94-efc3-4699-bfdf-0f38fcccf2fd","Type":"ContainerDied","Data":"62fe5060035496abe8e16f46641f1770147fa90c2bdcc0f3949f09a7f84677ae"}
Oct 30 17:33:39.937254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:39.937112    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62fe5060035496abe8e16f46641f1770147fa90c2bdcc0f3949f09a7f84677ae"
Oct 30 17:33:40.206163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:40.206085    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:40.904183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:40.904152    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:33:41.867814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:41.867777    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:43.671603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:43.671563    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:44.794411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:44.794359    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:44.907753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:44.907722    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:45.021628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:45.021595    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:45.359671 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:33:45.359623    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:33:45.653172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:45.653042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:45.769605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:45.769566    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:46.193439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.187830    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"]
Oct 30 17:33:46.193439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.188868    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3127ff94-efc3-4699-bfdf-0f38fcccf2fd" containerName="revoke-keys"
Oct 30 17:33:46.193439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.188889    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3127ff94-efc3-4699-bfdf-0f38fcccf2fd" containerName="revoke-keys"
Oct 30 17:33:46.193439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.189112    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3127ff94-efc3-4699-bfdf-0f38fcccf2fd" containerName="revoke-keys"
Oct 30 17:33:46.212307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.212276    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.273939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.273909    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"]
Oct 30 17:33:46.277518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.277489    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:46.312594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.312563    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:33:46.406662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.385128    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:46.406662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.387444    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vln8v\" (UniqueName: \"kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.406662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.387749    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.406662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.387825    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.543186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.543080    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vln8v\" (UniqueName: \"kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.545658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.545602    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.547776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.547746    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.578878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.578844    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:46.791798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.791763    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:47.032907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:46.964489    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vln8v\" (UniqueName: \"kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v\") pod \"maas-api-revoke-sub-2b029ea6-k2hw6\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:47.313725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:47.312914    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:47.586829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:47.586727    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:47.608532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:47.608486    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:47.875721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:47.862199    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:48.241726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.240752    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:48.325681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.322595    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:48.438815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.438771    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:33:48.482682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.465634    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:48.525921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.525876    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:48.598836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.598788    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:48.922728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.874513    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:33:48.922728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.876405    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:48.974850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.958597    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:33:48.974850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:48.958710    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:33:49.325011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.304727    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:49.325011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.304829    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:33:49.330852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.330793    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:33:49.523323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.363100    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:33:49.523323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.363486    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:33:49.523323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.465299    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:49.523323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.522299    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:49.554734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.535776    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:33:49.554734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.540087    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:49.554734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.554182    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:33:49.565902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.565824    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:33:49.566131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.566082    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:33:49.680690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.675891    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:49.987676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.979536    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:33:49.987676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.982270    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:33:49.987676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:49.983376    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:33:50.263857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.263768    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"]
Oct 30 17:33:50.391668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.383760    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:33:50.418308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.418250    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6" event={"ID":"28c64afe-863e-477d-8c56-305c6659ea04","Type":"ContainerStarted","Data":"0a8c3a660a8d8007be0b8192736e621854f9c1a3d1075f14a72f7387374bf15f"}
Oct 30 17:33:50.593950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.593886    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:50.638964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.638916    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:33:50.640881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.640844    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:50.675683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.661537    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:33:50.675683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.661808    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:50.675683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.661960    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:33:50.686234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.686187    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:50.802133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.785896    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:33:50.802133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.785966    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:33:50.915757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.885342    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9fcb39-pv5vl"]
Oct 30 17:33:50.997010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.996960    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:33:50.998625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:50.998482    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:51.235748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.232104    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"]
Oct 30 17:33:51.473698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.448891    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:51.600736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.532303    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": context deadline exceeded"
Oct 30 17:33:51.673992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.673945    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:33:51.676245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.676200    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:51.676558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.676536    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:33:51.682214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.682176    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:33:51.682471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.682449    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:33:51.682700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.682680    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:33:51.841613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.841056    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:51.851756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:51.851477    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 30 17:33:52.219968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.199422    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:33:52.221394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.221012    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:33:52.282903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.282863    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:52.485832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.485777    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:52.860730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.856760    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:52.911670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.905368    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:52.911670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.905550    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:52.911670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.905744    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:33:52.935843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:52.935795    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:53.094267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.094224    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651dd041-bd27-41dc-b84d-569438d1b955" path="/var/lib/kubelet/pods/651dd041-bd27-41dc-b84d-569438d1b955/volumes"
Oct 30 17:33:53.097500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.097452    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:53.394432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.394391    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.395175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.395143    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.395690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.395666    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.396941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.396886    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.397346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.397321    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.563716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.550748    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:53.723525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.699423    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:33:53.723901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.723836    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:53.760852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.760818    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"]
Oct 30 17:33:53.954153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.954103    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.982152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.982045    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:33:53.984322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:53.984270    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:54.008922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.008878    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:54.106145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.106091    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff628fa-htp5v"]
Oct 30 17:33:54.139798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.138312    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:54.526266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.526231    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:54.535080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.535048    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.541399    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e32c42-9eac-4e3b-a194-236431278de8" path="/var/lib/kubelet/pods/15e32c42-9eac-4e3b-a194-236431278de8/volumes"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542007    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542084    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542119    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542155    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542184    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542212    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:33:54.563722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.542239    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:33:54.566098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.566060    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:54.622142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.622096    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"]
Oct 30 17:33:54.634351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.634318    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:33:54.634534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.634378    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:33:54.634534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.634400    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"]
Oct 30 17:33:54.634534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.634433    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"]
Oct 30 17:33:54.635297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.635274    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.636799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.636779    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:33:54.639316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.639291    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9df3f8-8q7lw"]
Oct 30 17:33:54.741926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.741893    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.742120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.741994    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5pg\" (UniqueName: \"kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.742120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.742088    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.844106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.844072    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.844307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.844184    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5pg\" (UniqueName: \"kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.844393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.844336    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.845602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.845573    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.851038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.851014    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.935157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.935125    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5pg\" (UniqueName: \"kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg\") pod \"maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:54.957358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:54.957038    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:55.018258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:55.008674    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:55.502699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:55.501136    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"]
Oct 30 17:33:55.543839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:55.543787    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b" event={"ID":"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf","Type":"ContainerStarted","Data":"15895c58b2f5b5bfce27c17ca04aad6fa7e17814ee0d20522f5dc6d33e71c293"}
Oct 30 17:33:55.547438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:55.547412    5534 generic.go:356] "Generic (PLEG): container finished" podID="28c64afe-863e-477d-8c56-305c6659ea04" containerID="e382d8cf5d50f05ed9d84c67bad06322e072b060e20aabbea23c7f6fcc5be67e" exitCode=0
Oct 30 17:33:55.547438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:55.547446    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6" event={"ID":"28c64afe-863e-477d-8c56-305c6659ea04","Type":"ContainerDied","Data":"e382d8cf5d50f05ed9d84c67bad06322e072b060e20aabbea23c7f6fcc5be67e"}
Oct 30 17:33:56.396841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.396804    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042fc1c6-886d-4115-ac27-c200af8378e3" path="/var/lib/kubelet/pods/042fc1c6-886d-4115-ac27-c200af8378e3/volumes"
Oct 30 17:33:56.571211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.571168    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b" event={"ID":"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf","Type":"ContainerStarted","Data":"5681e9aee60d03b64eab5caa9363dcc2b8be0cee9742ab8f1e0891606a468228"}
Oct 30 17:33:56.779990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.779963    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:56.821961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.821904    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b" podStartSLOduration=3.821887993 podStartE2EDuration="3.821887993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:56.613885032 +0000 UTC m=+1242.375928845" watchObservedRunningTime="2026-10-30 17:33:56.821887993 +0000 UTC m=+1242.583931805"
Oct 30 17:33:56.921562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.921531    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca\") pod \"28c64afe-863e-477d-8c56-305c6659ea04\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") "
Oct 30 17:33:56.921772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.921621    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca\") pod \"28c64afe-863e-477d-8c56-305c6659ea04\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") "
Oct 30 17:33:56.921879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.921849    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v\" (UniqueName: \"kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v\") pod \"28c64afe-863e-477d-8c56-305c6659ea04\" (UID: \"28c64afe-863e-477d-8c56-305c6659ea04\") "
Oct 30 17:33:56.922003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.921968    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca" pod "28c64afe-863e-477d-8c56-305c6659ea04" (UID: "28c64afe-863e-477d-8c56-305c6659ea04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:56.922460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.922392    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:56.924022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.923992    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca" pod "28c64afe-863e-477d-8c56-305c6659ea04" (UID: "28c64afe-863e-477d-8c56-305c6659ea04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:56.924022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:56.924004    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v" pod "28c64afe-863e-477d-8c56-305c6659ea04" (UID: "28c64afe-863e-477d-8c56-305c6659ea04"). InnerVolumeSpecName "kube-api-access-vln8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:57.004105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.004015    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:57.023852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.023818    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vln8v\" (UniqueName: \"kubernetes.io/projected/28c64afe-863e-477d-8c56-305c6659ea04-kube-api-access-vln8v\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:57.023852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.023849    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c64afe-863e-477d-8c56-305c6659ea04-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:57.582739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.582688    5534 generic.go:356] "Generic (PLEG): container finished" podID="b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" containerID="5681e9aee60d03b64eab5caa9363dcc2b8be0cee9742ab8f1e0891606a468228" exitCode=0
Oct 30 17:33:57.583218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.582761    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b" event={"ID":"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf","Type":"ContainerDied","Data":"5681e9aee60d03b64eab5caa9363dcc2b8be0cee9742ab8f1e0891606a468228"}
Oct 30 17:33:57.584883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.584864    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"
Oct 30 17:33:57.584883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.584869    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6" event={"ID":"28c64afe-863e-477d-8c56-305c6659ea04","Type":"ContainerDied","Data":"0a8c3a660a8d8007be0b8192736e621854f9c1a3d1075f14a72f7387374bf15f"}
Oct 30 17:33:57.585086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:57.584916    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8c3a660a8d8007be0b8192736e621854f9c1a3d1075f14a72f7387374bf15f"
Oct 30 17:33:58.697522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.697490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:33:58.831750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.831718    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:58.967803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.967710    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca\") pod \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") "
Oct 30 17:33:58.968086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.967965    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg\" (UniqueName: \"kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg\") pod \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") "
Oct 30 17:33:58.968166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.968146    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca\") pod \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\" (UID: \"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf\") "
Oct 30 17:33:58.968234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.968152    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca" pod "b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" (UID: "b1f47432-1ab1-4f8e-9f24-61b1ccb474cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:58.968751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.968726    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:58.970724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.970692    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg" pod "b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" (UID: "b1f47432-1ab1-4f8e-9f24-61b1ccb474cf"). InnerVolumeSpecName "kube-api-access-ml5pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:58.971014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:58.970994    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca" pod "b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" (UID: "b1f47432-1ab1-4f8e-9f24-61b1ccb474cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:59.070318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:59.070282    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:59.070318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:59.070310    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5pg\" (UniqueName: \"kubernetes.io/projected/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf-kube-api-access-ml5pg\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:33:59.606669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:59.606628    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"
Oct 30 17:33:59.606961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:59.606688    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b" event={"ID":"b1f47432-1ab1-4f8e-9f24-61b1ccb474cf","Type":"ContainerDied","Data":"15895c58b2f5b5bfce27c17ca04aad6fa7e17814ee0d20522f5dc6d33e71c293"}
Oct 30 17:33:59.607077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:33:59.606994    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15895c58b2f5b5bfce27c17ca04aad6fa7e17814ee0d20522f5dc6d33e71c293"
Oct 30 17:34:00.392201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.392164    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:00.973840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.973754    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"]
Oct 30 17:34:00.974557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.974524    5534 cpu_manager.go:414] "containerMap: removing container" podUID="28c64afe-863e-477d-8c56-305c6659ea04" containerName="revoke-keys"
Oct 30 17:34:00.974557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.974548    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c64afe-863e-477d-8c56-305c6659ea04" containerName="revoke-keys"
Oct 30 17:34:00.974761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.974603    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" containerName="revoke-keys"
Oct 30 17:34:00.974761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.974612    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" containerName="revoke-keys"
Oct 30 17:34:00.974860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.974827    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" containerName="revoke-keys"
Oct 30 17:34:00.974860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.974850    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c64afe-863e-477d-8c56-305c6659ea04" containerName="revoke-keys"
Oct 30 17:34:00.977539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:00.977518    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.012189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.012161    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"]
Oct 30 17:34:01.023827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.023757    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:34:01.024891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.024871    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:34:01.101128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.101089    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.101418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.101390    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bww9r\" (UniqueName: \"kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.101535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.101515    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.202911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.202874    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww9r\" (UniqueName: \"kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.203139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.203057    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.203139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.203108    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.204268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.204211    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.206742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.206699    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.227714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.227683    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww9r\" (UniqueName: \"kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r\") pod \"maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.293598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.293511    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:01.612944 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:34:01.612912    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0278d23_7d73_468f_90f1_84e1b72ba9f3.slice/crio-1f49798d1fe7c8fdd7a6f0d9d5ff35dff5a243612ea58ac6f644ac1e0dcbe909 WatchSource:0}: Error finding container 1f49798d1fe7c8fdd7a6f0d9d5ff35dff5a243612ea58ac6f644ac1e0dcbe909: Status 404 returned error can't find the container with id 1f49798d1fe7c8fdd7a6f0d9d5ff35dff5a243612ea58ac6f644ac1e0dcbe909
Oct 30 17:34:01.613325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.613280    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"]
Oct 30 17:34:01.640722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.640687    5534 generic.go:356] "Generic (PLEG): container finished" podID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerID="87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc" exitCode=0
Oct 30 17:34:01.640965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.640930    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" event={"ID":"bac9d291-b3ce-48a4-bf07-8b36709a01ff","Type":"ContainerDied","Data":"87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"}
Oct 30 17:34:01.641523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.641502    5534 scope.go:122] "RemoveContainer" containerID="87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"
Oct 30 17:34:01.643443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.643409    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:01.643558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.643505    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:01.643623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.643555    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:01.645189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:01.645159    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x" event={"ID":"d0278d23-7d73-468f-90f1-84e1b72ba9f3","Type":"ContainerStarted","Data":"1f49798d1fe7c8fdd7a6f0d9d5ff35dff5a243612ea58ac6f644ac1e0dcbe909"}
Oct 30 17:34:02.033095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.033055    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"]
Oct 30 17:34:02.038138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.038107    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b56010-njlgq"]
Oct 30 17:34:02.390215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.390180    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0044f242-b3e3-44d6-897b-c147c1ff6118" path="/var/lib/kubelet/pods/0044f242-b3e3-44d6-897b-c147c1ff6118/volumes"
Oct 30 17:34:02.406556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.406518    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:02.679754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.677098    5534 generic.go:356] "Generic (PLEG): container finished" podID="d0278d23-7d73-468f-90f1-84e1b72ba9f3" containerID="7ac41ead9489155d69b56cedad040f64a1d28db9bf6beff60113108271e8367e" exitCode=0
Oct 30 17:34:02.679754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.677319    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x" event={"ID":"d0278d23-7d73-468f-90f1-84e1b72ba9f3","Type":"ContainerDied","Data":"7ac41ead9489155d69b56cedad040f64a1d28db9bf6beff60113108271e8367e"}
Oct 30 17:34:02.688703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:02.682103    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" event={"ID":"bac9d291-b3ce-48a4-bf07-8b36709a01ff","Type":"ContainerStarted","Data":"b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656"}
Oct 30 17:34:03.307731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:03.307688    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:34:03.309177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:03.309146    5534 patch_prober.go:29] interesting pod/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 17:34:03.309304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:03.309217    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 17:34:03.942334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:03.942309    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:04.041525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.041089    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"]
Oct 30 17:34:04.048746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.048697    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33218e82-jgxx5"]
Oct 30 17:34:04.084709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.066215    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca\") pod \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") "
Oct 30 17:34:04.084709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.066352    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca\") pod \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") "
Oct 30 17:34:04.084709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.066699    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r\" (UniqueName: \"kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r\") pod \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\" (UID: \"d0278d23-7d73-468f-90f1-84e1b72ba9f3\") "
Oct 30 17:34:04.084709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.077409    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca" pod "d0278d23-7d73-468f-90f1-84e1b72ba9f3" (UID: "d0278d23-7d73-468f-90f1-84e1b72ba9f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:04.087902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.087843    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca" pod "d0278d23-7d73-468f-90f1-84e1b72ba9f3" (UID: "d0278d23-7d73-468f-90f1-84e1b72ba9f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:04.092201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.092161    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r" pod "d0278d23-7d73-468f-90f1-84e1b72ba9f3" (UID: "d0278d23-7d73-468f-90f1-84e1b72ba9f3"). InnerVolumeSpecName "kube-api-access-bww9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:04.180132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.180090    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bww9r\" (UniqueName: \"kubernetes.io/projected/d0278d23-7d73-468f-90f1-84e1b72ba9f3-kube-api-access-bww9r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:04.180383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.180366    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:04.180502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.180487    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0278d23-7d73-468f-90f1-84e1b72ba9f3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:04.213466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.213427    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:04.308468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.308379    5534 patch_prober.go:29] interesting pod/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 17:34:04.308468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.308449    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 17:34:04.404178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.404146    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94229a0-5c04-40c6-9c12-bd769ec9ad2f" path="/var/lib/kubelet/pods/f94229a0-5c04-40c6-9c12-bd769ec9ad2f/volumes"
Oct 30 17:34:04.614410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.611563    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:04.722336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.722298    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"
Oct 30 17:34:04.722795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.722398    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x" event={"ID":"d0278d23-7d73-468f-90f1-84e1b72ba9f3","Type":"ContainerDied","Data":"1f49798d1fe7c8fdd7a6f0d9d5ff35dff5a243612ea58ac6f644ac1e0dcbe909"}
Oct 30 17:34:04.722795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.722457    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f49798d1fe7c8fdd7a6f0d9d5ff35dff5a243612ea58ac6f644ac1e0dcbe909"
Oct 30 17:34:04.803663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:04.796467    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:05.051763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:05.048744    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"]
Oct 30 17:34:05.057665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:05.055689    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"]
Oct 30 17:34:05.081667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:05.070687    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59c5757-qzvjg"]
Oct 30 17:34:05.108670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:05.097319    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c10df0-f7vlk"]
Oct 30 17:34:05.309377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:05.308535    5534 patch_prober.go:29] interesting pod/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 17:34:05.309377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:05.308695    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 17:34:06.308719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:06.308678    5534 patch_prober.go:29] interesting pod/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 17:34:06.309139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:06.308761    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 17:34:06.380670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:06.378477    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:06.414608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:06.410122    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae" path="/var/lib/kubelet/pods/af0ff7de-4dd2-449d-9fc8-5adcfc19b8ae/volumes"
Oct 30 17:34:06.420668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:06.418516    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb577c62-ebe9-4661-a5c4-cdcce87c9a80" path="/var/lib/kubelet/pods/eb577c62-ebe9-4661-a5c4-cdcce87c9a80/volumes"
Oct 30 17:34:06.635294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:06.635256    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:34:07.307813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:07.307778    5534 patch_prober.go:29] interesting pod/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 17:34:07.307993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:07.307869    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 17:34:08.313853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.313818    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:34:08.315049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.315026    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:34:08.319739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.319714    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:08.995028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.992171    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"]
Oct 30 17:34:08.995028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.993285    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d0278d23-7d73-468f-90f1-84e1b72ba9f3" containerName="revoke-keys"
Oct 30 17:34:08.995028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.993309    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0278d23-7d73-468f-90f1-84e1b72ba9f3" containerName="revoke-keys"
Oct 30 17:34:08.995028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.993598    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0278d23-7d73-468f-90f1-84e1b72ba9f3" containerName="revoke-keys"
Oct 30 17:34:08.997764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:08.997741    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.028571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.028544    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"]
Oct 30 17:34:09.123261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123226    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123277    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123445    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123584    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn64r\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123673    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123785    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123815    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123836    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.123874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.123865    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.224770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.224732    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.224934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.224846    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.224934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.224889    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.224954    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225194    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn64r\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225223    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225291    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225363    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225388    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225417    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.225959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225775    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.226046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.226022    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.226046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225806    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.226155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.225951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.227603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.227578    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.227841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.227820    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.249151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.249076    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.249317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.249296    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn64r\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.312153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.312118    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:09.647295 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:34:09.647254    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61511210_6491_4043_b94a_f24550e42e66.slice/crio-012aacaeb651c7a134a5efb056cccfce84023507ce904b92dce4c81b9eb428fd WatchSource:0}: Error finding container 012aacaeb651c7a134a5efb056cccfce84023507ce904b92dce4c81b9eb428fd: Status 404 returned error can't find the container with id 012aacaeb651c7a134a5efb056cccfce84023507ce904b92dce4c81b9eb428fd
Oct 30 17:34:09.648637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.648613    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"]
Oct 30 17:34:09.650041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.650010    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:09.650110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.650079    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:09.650149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.650110    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:09.777265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.777214    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" event={"ID":"61511210-6491-4043-b94a-f24550e42e66","Type":"ContainerStarted","Data":"26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76"}
Oct 30 17:34:09.777502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.777293    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" event={"ID":"61511210-6491-4043-b94a-f24550e42e66","Type":"ContainerStarted","Data":"012aacaeb651c7a134a5efb056cccfce84023507ce904b92dce4c81b9eb428fd"}
Oct 30 17:34:09.828839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:09.828773    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podStartSLOduration=1.828748686 podStartE2EDuration="1.828748686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:09.822168511 +0000 UTC m=+1255.584212337" watchObservedRunningTime="2026-10-30 17:34:09.828748686 +0000 UTC m=+1255.590792503"
Oct 30 17:34:10.130627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.130577    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:10.180674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.179876    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"]
Oct 30 17:34:10.187144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.187114    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.203239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.197851    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"]
Oct 30 17:34:10.203239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.203001    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:34:10.203239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.203198    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:34:10.245764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.245720    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4m2\" (UniqueName: \"kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.246048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.246022    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.246481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.246254    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.313123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.313089    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:10.315141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.315116    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:10.315274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.315166    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:10.350712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.350678    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.350876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.350806    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4m2\" (UniqueName: \"kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.350921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.350900    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.351309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.351287    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.353390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.353371    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.374433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.374407    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4m2\" (UniqueName: \"kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2\") pod \"maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.518030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.517929    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:10.911264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:10.911229    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"]
Oct 30 17:34:11.313145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.313108    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:11.313335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.313176    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:11.632456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.632410    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:11.751728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.748675    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:11.850658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.850598    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk" event={"ID":"1add579f-cd2e-4be3-bf73-444679a4a97c","Type":"ContainerStarted","Data":"caee496daea8d78d94d77fd507cbbae48cdfc80c65e646d0e9b8e95671a8b448"}
Oct 30 17:34:11.850904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.850886    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk" event={"ID":"1add579f-cd2e-4be3-bf73-444679a4a97c","Type":"ContainerStarted","Data":"84ed59a7ba7e658b0b6c3014dfe9002ed9181fcb2042f045500fbbf81ae5994f"}
Oct 30 17:34:11.944570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:11.944417    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:12.101413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.101382    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:12.315724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.315050    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:12.315724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.315110    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:12.762473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.762409    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:12.893057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.893000    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:12.895104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.895076    5534 generic.go:356] "Generic (PLEG): container finished" podID="1add579f-cd2e-4be3-bf73-444679a4a97c" containerID="caee496daea8d78d94d77fd507cbbae48cdfc80c65e646d0e9b8e95671a8b448" exitCode=0
Oct 30 17:34:12.895316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.895297    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk" event={"ID":"1add579f-cd2e-4be3-bf73-444679a4a97c","Type":"ContainerDied","Data":"caee496daea8d78d94d77fd507cbbae48cdfc80c65e646d0e9b8e95671a8b448"}
Oct 30 17:34:12.929074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.929006    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk" podStartSLOduration=2.9289875419999998 podStartE2EDuration="2.928987542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:11.916017317 +0000 UTC m=+1257.678061140" watchObservedRunningTime="2026-10-30 17:34:12.928987542 +0000 UTC m=+1258.691031356"
Oct 30 17:34:12.936598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.936567    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"]
Oct 30 17:34:12.943103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:12.943070    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8ac7b58-wqj9b"]
Oct 30 17:34:13.076107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.074236    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:13.149707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.149667    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"]
Oct 30 17:34:13.150708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.150669    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy" containerID="cri-o://b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656" gracePeriod=30
Oct 30 17:34:13.219348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.219309    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:13.343730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.342369    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:13.343730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.342428    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:13.794923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:13.794811    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:14.148842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.148819    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:14.239992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.239964    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2\" (UniqueName: \"kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2\") pod \"1add579f-cd2e-4be3-bf73-444679a4a97c\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") "
Oct 30 17:34:14.240218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.240202    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca\") pod \"1add579f-cd2e-4be3-bf73-444679a4a97c\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") "
Oct 30 17:34:14.240386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.240346    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca\") pod \"1add579f-cd2e-4be3-bf73-444679a4a97c\" (UID: \"1add579f-cd2e-4be3-bf73-444679a4a97c\") "
Oct 30 17:34:14.244747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.244719    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca" pod "1add579f-cd2e-4be3-bf73-444679a4a97c" (UID: "1add579f-cd2e-4be3-bf73-444679a4a97c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:14.245217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.245192    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2" pod "1add579f-cd2e-4be3-bf73-444679a4a97c" (UID: "1add579f-cd2e-4be3-bf73-444679a4a97c"). InnerVolumeSpecName "kube-api-access-sq4m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:14.251207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.251183    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca" pod "1add579f-cd2e-4be3-bf73-444679a4a97c" (UID: "1add579f-cd2e-4be3-bf73-444679a4a97c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:14.299705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.286169    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:14.316504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.316455    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:14.316700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.316534    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:14.342701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.342667    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:14.342701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.342703    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1add579f-cd2e-4be3-bf73-444679a4a97c-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:14.342885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.342719    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq4m2\" (UniqueName: \"kubernetes.io/projected/1add579f-cd2e-4be3-bf73-444679a4a97c-kube-api-access-sq4m2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:14.627722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.627688    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f47432-1ab1-4f8e-9f24-61b1ccb474cf" path="/var/lib/kubelet/pods/b1f47432-1ab1-4f8e-9f24-61b1ccb474cf/volumes"
Oct 30 17:34:14.896418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.896023    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:14.936355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.936326    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"
Oct 30 17:34:14.936355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.936344    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk" event={"ID":"1add579f-cd2e-4be3-bf73-444679a4a97c","Type":"ContainerDied","Data":"84ed59a7ba7e658b0b6c3014dfe9002ed9181fcb2042f045500fbbf81ae5994f"}
Oct 30 17:34:14.936562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:14.936393    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ed59a7ba7e658b0b6c3014dfe9002ed9181fcb2042f045500fbbf81ae5994f"
Oct 30 17:34:15.248433 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:34:15.248352    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:34:15.256667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:15.255961    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:15.312813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:15.312769    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:15.312966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:15.312820    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:16.313115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:16.313084    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:16.313519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:16.313166    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:17.068535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:17.068498    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:17.312679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:17.312619    5534 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 17:34:17.312850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:17.312716    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 17:34:17.801142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:17.801093    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:17.934361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:17.934320    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:18.071625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.071530    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"]
Oct 30 17:34:18.078873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.078841    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e8bf1-74zgv"]
Oct 30 17:34:18.317892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.317867    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:18.318351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.318329    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:18.319098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.319082    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:34:18.373122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.373097    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9" path="/var/lib/kubelet/pods/d2e8ed3e-0e89-4fb3-83cf-be44cd5374d9/volumes"
Oct 30 17:34:18.502670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.502630    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:34:18.629722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.629599    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.629722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.629671    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.629722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.629712    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.629781    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.629920    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.629979    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630067    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630116    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630140    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:18.630261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630223    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:18.630366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630282    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data\") pod \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\" (UID: \"bac9d291-b3ce-48a4-bf07-8b36709a01ff\") "
Oct 30 17:34:18.630419 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630384    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:18.630491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630472    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:18.630709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.630657    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:18.631138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.631114    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.631138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.631140    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.631266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.631149    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.631266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.631158    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.631266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.631166    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.632942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.632913    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:34:18.633025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.632951    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "kube-api-access-7frff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:18.633025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.633006    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:18.633740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.633721    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy" pod "bac9d291-b3ce-48a4-bf07-8b36709a01ff" (UID: "bac9d291-b3ce-48a4-bf07-8b36709a01ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:18.731527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.731486    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"]
Oct 30 17:34:18.752222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.731807    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt" podUID="4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" containerName="maas-api" containerID="cri-o://426753c768c6773b25bbec36c0e06bea3dfa54659e33b688a0845388ebeb59ea" gracePeriod=30
Oct 30 17:34:18.752222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.732889    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.752222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.732918    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frff\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-kube-api-access-7frff\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.752222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.732937    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.752222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.732951    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac9d291-b3ce-48a4-bf07-8b36709a01ff-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:18.764082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.764043    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:18.874852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:18.874819    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:19.040772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.040735    5534 generic.go:356] "Generic (PLEG): container finished" podID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerID="b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656" exitCode=0
Oct 30 17:34:19.041304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.041276    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" event={"ID":"bac9d291-b3ce-48a4-bf07-8b36709a01ff","Type":"ContainerDied","Data":"b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656"}
Oct 30 17:34:19.041477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.041461    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2" event={"ID":"bac9d291-b3ce-48a4-bf07-8b36709a01ff","Type":"ContainerDied","Data":"a94518c27eecf860e468df3e2b047fd370e3c7a0802d363d8df41f4d3739411d"}
Oct 30 17:34:19.041601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.041587    5534 scope.go:122] "RemoveContainer" containerID="b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656"
Oct 30 17:34:19.041900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.041882    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"
Oct 30 17:34:19.092983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.092950    5534 generic.go:356] "Generic (PLEG): container finished" podID="4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" containerID="426753c768c6773b25bbec36c0e06bea3dfa54659e33b688a0845388ebeb59ea" exitCode=0
Oct 30 17:34:19.093991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.093965    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt" event={"ID":"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a","Type":"ContainerDied","Data":"426753c768c6773b25bbec36c0e06bea3dfa54659e33b688a0845388ebeb59ea"}
Oct 30 17:34:19.101938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.101898    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:34:19.111884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.111861    5534 scope.go:122] "RemoveContainer" containerID="87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"
Oct 30 17:34:19.150666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.139730    5534 scope.go:122] "RemoveContainer" containerID="b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656"
Oct 30 17:34:19.150666 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:34:19.141103    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656\": container with ID starting with b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656 not found: ID does not exist" containerID="b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656"
Oct 30 17:34:19.150666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.141147    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656"} err="failed to get container status \"b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656\": rpc error: code = NotFound desc = could not find container \"b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656\": container with ID starting with b2110b538ec7af0a4a2e0a2665f19df78e5cd81a51816be124dd04142933f656 not found: ID does not exist"
Oct 30 17:34:19.150666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.141180    5534 scope.go:122] "RemoveContainer" containerID="87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"
Oct 30 17:34:19.150666 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:34:19.141684    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc\": container with ID starting with 87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc not found: ID does not exist" containerID="87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"
Oct 30 17:34:19.150666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.141722    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc"} err="failed to get container status \"87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc\": rpc error: code = NotFound desc = could not find container \"87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc\": container with ID starting with 87ea79cb73c7e8627fdab4ca7a8b906052ca3cf3aaa3928548d5135e835b4edc not found: ID does not exist"
Oct 30 17:34:19.267669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.242741    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs\") pod \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") "
Oct 30 17:34:19.267669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.242954    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls\") pod \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") "
Oct 30 17:34:19.267669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.243067    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl\" (UniqueName: \"kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl\") pod \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") "
Oct 30 17:34:19.267669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.243096    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca\") pod \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\" (UID: \"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a\") "
Oct 30 17:34:19.267669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.250723    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca" pod "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" (UID: "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:19.267669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.251261    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"]
Oct 30 17:34:19.363751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.354075    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:19.384724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.378576    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f8ac7b58-openshift-default-764f68897c-lmgp2"]
Oct 30 17:34:19.415926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.415874    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl" pod "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" (UID: "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a"). InnerVolumeSpecName "kube-api-access-rzgwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:19.424241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.424189    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls" pod "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" (UID: "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:19.424957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.424884    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs" pod "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" (UID: "4b62c09c-e2ad-4830-b9c0-0b7cc831e31a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:19.457400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.456728    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:19.457400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.457068    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:19.457400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.457096    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgwl\" (UniqueName: \"kubernetes.io/projected/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a-kube-api-access-rzgwl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:19.607537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.607496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:19.734068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.734026    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:19.765227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.765155    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"]
Oct 30 17:34:19.772605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.772492    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-84af0117-d4h7x"]
Oct 30 17:34:19.864730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.864609    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:19.875965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.875934    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"]
Oct 30 17:34:19.877241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877216    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy"
Oct 30 17:34:19.877241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877242    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy"
Oct 30 17:34:19.877461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877269    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy"
Oct 30 17:34:19.877461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877278    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy"
Oct 30 17:34:19.877461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877320    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1add579f-cd2e-4be3-bf73-444679a4a97c" containerName="revoke-keys"
Oct 30 17:34:19.877461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877338    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1add579f-cd2e-4be3-bf73-444679a4a97c" containerName="revoke-keys"
Oct 30 17:34:19.877461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877360    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" containerName="maas-api"
Oct 30 17:34:19.877461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877369    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" containerName="maas-api"
Oct 30 17:34:19.877806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877558    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1add579f-cd2e-4be3-bf73-444679a4a97c" containerName="revoke-keys"
Oct 30 17:34:19.877806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877577    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" containerName="maas-api"
Oct 30 17:34:19.877806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.877594    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy"
Oct 30 17:34:19.881526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.881493    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:19.915716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.906967    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-86c97ce8\""
Oct 30 17:34:19.915716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.914709    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"]
Oct 30 17:34:19.915716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.915549    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-86c97ce8-dockercfg-6qxnf\""
Oct 30 17:34:19.972110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.972046    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4cf\" (UniqueName: \"kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf\") pod \"payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:19.973986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.973945    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:19.983528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.983115    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"]
Oct 30 17:34:19.985304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:19.984736    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" containerName="istio-proxy"
Oct 30 17:34:20.000087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.000055    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:20.003949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.003917    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.004395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.004372    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"]
Oct 30 17:34:20.105665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.085012    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4cf\" (UniqueName: \"kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf\") pod \"payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:20.105665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.085163    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:20.105665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.085244    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") " pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.105665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.085335    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4xr\" (UniqueName: \"kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr\") pod \"payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") " pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.113675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.106595    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:20.189593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.189495    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4xr\" (UniqueName: \"kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr\") pod \"payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") " pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.195096 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.195040    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4cf\" (UniqueName: \"kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf\") pod \"payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:20.226661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.226504    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:20.229634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.229597    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt" event={"ID":"4b62c09c-e2ad-4830-b9c0-0b7cc831e31a","Type":"ContainerDied","Data":"c79f5f545eb309a7eae2c1b44b246e7c5e63f53ad7f77b803ad8873d11cd5dc8"}
Oct 30 17:34:20.229914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.229894    5534 scope.go:122] "RemoveContainer" containerID="426753c768c6773b25bbec36c0e06bea3dfa54659e33b688a0845388ebeb59ea"
Oct 30 17:34:20.230166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.230147    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"
Oct 30 17:34:20.271105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.256950    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4xr\" (UniqueName: \"kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr\") pod \"payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") " pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.293497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.293455    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") " pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.294495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.294308    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") " pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.311676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.309336    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"]
Oct 30 17:34:20.333848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.333820    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-84af0117-79d8c75d49-764tt"]
Oct 30 17:34:20.349676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.349625    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:20.399046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.398521    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b62c09c-e2ad-4830-b9c0-0b7cc831e31a" path="/var/lib/kubelet/pods/4b62c09c-e2ad-4830-b9c0-0b7cc831e31a/volumes"
Oct 30 17:34:20.399808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.399242    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac9d291-b3ce-48a4-bf07-8b36709a01ff" path="/var/lib/kubelet/pods/bac9d291-b3ce-48a4-bf07-8b36709a01ff/volumes"
Oct 30 17:34:20.400004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.399893    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0278d23-7d73-468f-90f1-84e1b72ba9f3" path="/var/lib/kubelet/pods/d0278d23-7d73-468f-90f1-84e1b72ba9f3/volumes"
Oct 30 17:34:20.642111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.642077    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"]
Oct 30 17:34:20.877594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.877323    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"]
Oct 30 17:34:20.956798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:20.956175    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:21.077813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.073809    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:21.273946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.273865    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:21.303258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.303221    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk" event={"ID":"0d5865a5-c848-48a8-a939-ca628b8f71a8","Type":"ContainerStarted","Data":"3031286eee5003027ffba8ba57cfd5c35806c6452bb1d9a65f2f9ba1f8cdfd5f"}
Oct 30 17:34:21.303519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.303498    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk" event={"ID":"0d5865a5-c848-48a8-a939-ca628b8f71a8","Type":"ContainerStarted","Data":"218e5afb2a445c6aca7061e661b5a2d54dba64f948beb99143584e4d08391865"}
Oct 30 17:34:21.304841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.304811    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:21.307079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.307054    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp" event={"ID":"fdf87969-a5c2-43f1-bbb8-5a837f9b6780","Type":"ContainerStarted","Data":"e43802a23be9573503a34ad281ec681606ccd2064bdc0c750f705c2ede4ad8cf"}
Oct 30 17:34:21.307239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.307101    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp" event={"ID":"fdf87969-a5c2-43f1-bbb8-5a837f9b6780","Type":"ContainerStarted","Data":"44ac631e3c7c7d6845ebce9bad6ceadfc68a0699c8c6564beaee8e9bb85e6d4a"}
Oct 30 17:34:21.307856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.307841    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:21.356395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.356341    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk" podStartSLOduration=2.356325315 podStartE2EDuration="2.356325315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:21.349818313 +0000 UTC m=+1267.111862126" watchObservedRunningTime="2026-10-30 17:34:21.356325315 +0000 UTC m=+1267.118369129"
Oct 30 17:34:21.393130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.393030    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp" podStartSLOduration=2.393012988 podStartE2EDuration="2.393012988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:21.389445729 +0000 UTC m=+1267.151489556" watchObservedRunningTime="2026-10-30 17:34:21.393012988 +0000 UTC m=+1267.155056802"
Oct 30 17:34:21.410883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.410852    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:21.983930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:21.980551    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:23.616002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:23.615889    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:24.283685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:24.276519    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:24.445747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:24.443386    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:24.516671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:24.512323    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"]
Oct 30 17:34:24.525789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:24.524119    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15a954ad-n5nkk"]
Oct 30 17:34:24.578441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:24.578315    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:24.709324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:24.708416    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:25.119667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.115207    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"]
Oct 30 17:34:25.142687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.139556    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.155043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.154997    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.155232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.155054    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6df\" (UniqueName: \"kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.155232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.155212    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.155850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.155830    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"]
Oct 30 17:34:25.218859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.218206    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:34:25.218859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.218280    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:34:25.270722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.268879    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.270722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.268930    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6df\" (UniqueName: \"kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.270722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.269112    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.270722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.270199    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.274699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.273856    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.292816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.292789    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6df\" (UniqueName: \"kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df\") pod \"maas-api-revoke-sub-a72dd180-2724r\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.510864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.510780    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:25.691769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.691690    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:25.809586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.809513    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:25.823879 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:34:25.823853    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ce3963_ebe2_4fbb_a4d6_5befc42b96d0.slice/crio-d0c164f145a68ac04711d385fb14369efc5e837da9cdc87ddd36e42e8d6f4a03 WatchSource:0}: Error finding container d0c164f145a68ac04711d385fb14369efc5e837da9cdc87ddd36e42e8d6f4a03: Status 404 returned error can't find the container with id d0c164f145a68ac04711d385fb14369efc5e837da9cdc87ddd36e42e8d6f4a03
Oct 30 17:34:25.825496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:25.825467    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"]
Oct 30 17:34:26.373725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:26.373693    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1add579f-cd2e-4be3-bf73-444679a4a97c" path="/var/lib/kubelet/pods/1add579f-cd2e-4be3-bf73-444679a4a97c/volumes"
Oct 30 17:34:26.441817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:26.441785    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r" event={"ID":"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0","Type":"ContainerStarted","Data":"559bc3c4da267dd1df4bbd1938aa657e0589d1303462439bde8b5a21006abe20"}
Oct 30 17:34:26.441954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:26.441839    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r" event={"ID":"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0","Type":"ContainerStarted","Data":"d0c164f145a68ac04711d385fb14369efc5e837da9cdc87ddd36e42e8d6f4a03"}
Oct 30 17:34:26.484671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:26.484607    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r" podStartSLOduration=1.48459371 podStartE2EDuration="1.48459371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:26.481250864 +0000 UTC m=+1272.243294681" watchObservedRunningTime="2026-10-30 17:34:26.48459371 +0000 UTC m=+1272.246637523"
Oct 30 17:34:26.698855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:26.698813    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:27.452267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:27.452236    5534 generic.go:356] "Generic (PLEG): container finished" podID="08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" containerID="559bc3c4da267dd1df4bbd1938aa657e0589d1303462439bde8b5a21006abe20" exitCode=0
Oct 30 17:34:27.452761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:27.452321    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r" event={"ID":"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0","Type":"ContainerDied","Data":"559bc3c4da267dd1df4bbd1938aa657e0589d1303462439bde8b5a21006abe20"}
Oct 30 17:34:28.034488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.034455    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"]
Oct 30 17:34:28.039509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.039487    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eadb3ac-xxd78"]
Oct 30 17:34:28.372784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.372746    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a8d3d5-15d8-4e09-b9af-f61c3f571336" path="/var/lib/kubelet/pods/90a8d3d5-15d8-4e09-b9af-f61c3f571336/volumes"
Oct 30 17:34:28.495769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.495726    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:28.692881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.692851    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:28.745475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.745438    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca\") pod \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") "
Oct 30 17:34:28.745753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.745542    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df\" (UniqueName: \"kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df\") pod \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") "
Oct 30 17:34:28.745903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.745879    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca\") pod \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\" (UID: \"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0\") "
Oct 30 17:34:28.745903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.745891    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca" pod "08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" (UID: "08ce3963-ebe2-4fbb-a4d6-5befc42b96d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:28.746833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.746804    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:28.748507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.748460    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df" pod "08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" (UID: "08ce3963-ebe2-4fbb-a4d6-5befc42b96d0"). InnerVolumeSpecName "kube-api-access-wv6df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:28.749598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.749554    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca" pod "08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" (UID: "08ce3963-ebe2-4fbb-a4d6-5befc42b96d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:28.848244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.848197    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6df\" (UniqueName: \"kubernetes.io/projected/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-kube-api-access-wv6df\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:28.848244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:28.848240    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:29.483371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:29.483335    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r" event={"ID":"08ce3963-ebe2-4fbb-a4d6-5befc42b96d0","Type":"ContainerDied","Data":"d0c164f145a68ac04711d385fb14369efc5e837da9cdc87ddd36e42e8d6f4a03"}
Oct 30 17:34:29.483576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:29.483387    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c164f145a68ac04711d385fb14369efc5e837da9cdc87ddd36e42e8d6f4a03"
Oct 30 17:34:29.483576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:29.483342    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"
Oct 30 17:34:29.500876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:29.500848    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:30.207679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:30.207609    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:32.013251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:32.013216    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:32.353609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:32.353581    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:34:33.032206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:33.032170    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"]
Oct 30 17:34:33.044034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:33.044005    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd1f492-w58vs"]
Oct 30 17:34:33.367382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:33.367349    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:34:33.686818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:33.686632    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:33.802831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:33.802794    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:33.921636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:33.921602    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:34.371825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:34.371778    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9dfa281-33d8-476a-a736-fa65f964602a" path="/var/lib/kubelet/pods/f9dfa281-33d8-476a-a736-fa65f964602a/volumes"
Oct 30 17:34:35.484126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.484086    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:35.535043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.534966    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"]
Oct 30 17:34:35.536981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.536952    5534 cpu_manager.go:414] "containerMap: removing container" podUID="08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" containerName="revoke-keys"
Oct 30 17:34:35.536981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.536981    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" containerName="revoke-keys"
Oct 30 17:34:35.537494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.537472    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" containerName="revoke-keys"
Oct 30 17:34:35.551975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.551945    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.559492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.557264    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"]
Oct 30 17:34:35.584779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.584755    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:34:35.584949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.584755    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:34:35.673494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.673326    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smclw\" (UniqueName: \"kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.673494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.673453    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.673494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.673493    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.774627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.774516    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.774627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.774575    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.774912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.774818    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smclw\" (UniqueName: \"kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.775336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.775307    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.777334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.777309    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.798342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.798303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smclw\" (UniqueName: \"kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw\") pod \"maas-api-revoke-sub-04d047c6-6vxmh\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:35.882069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:35.882034    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:36.202373 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:34:36.201555    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode396bd00_76e2_4db9_97fc_4d65d04b4218.slice/crio-1c1cdabf281b86c1cab0e9f25769880977ff4c4e90abda09814f7de448ef9c6c WatchSource:0}: Error finding container 1c1cdabf281b86c1cab0e9f25769880977ff4c4e90abda09814f7de448ef9c6c: Status 404 returned error can't find the container with id 1c1cdabf281b86c1cab0e9f25769880977ff4c4e90abda09814f7de448ef9c6c
Oct 30 17:34:36.203810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:36.203784    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"]
Oct 30 17:34:36.620691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:36.620633    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:36.630075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:36.630032    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh" event={"ID":"e396bd00-76e2-4db9-97fc-4d65d04b4218","Type":"ContainerStarted","Data":"159b55a66c4ff45489c2a8fcdd27ebe5bd709929958bc55fbc23aa531a42e566"}
Oct 30 17:34:36.630288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:36.630098    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh" event={"ID":"e396bd00-76e2-4db9-97fc-4d65d04b4218","Type":"ContainerStarted","Data":"1c1cdabf281b86c1cab0e9f25769880977ff4c4e90abda09814f7de448ef9c6c"}
Oct 30 17:34:36.680186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:36.680123    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh" podStartSLOduration=1.680103157 podStartE2EDuration="1.680103157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:36.671479273 +0000 UTC m=+1282.433523086" watchObservedRunningTime="2026-10-30 17:34:36.680103157 +0000 UTC m=+1282.442146971"
Oct 30 17:34:37.294577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:37.294546    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:37.420198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:37.420160    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:37.640332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:37.640288    5534 generic.go:356] "Generic (PLEG): container finished" podID="e396bd00-76e2-4db9-97fc-4d65d04b4218" containerID="159b55a66c4ff45489c2a8fcdd27ebe5bd709929958bc55fbc23aa531a42e566" exitCode=0
Oct 30 17:34:37.640812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:37.640342    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh" event={"ID":"e396bd00-76e2-4db9-97fc-4d65d04b4218","Type":"ContainerDied","Data":"159b55a66c4ff45489c2a8fcdd27ebe5bd709929958bc55fbc23aa531a42e566"}
Oct 30 17:34:37.724773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:37.724678    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:38.816510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.816485    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:38.922487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.922388    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca\") pod \"e396bd00-76e2-4db9-97fc-4d65d04b4218\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") "
Oct 30 17:34:38.922698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.922599    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca\") pod \"e396bd00-76e2-4db9-97fc-4d65d04b4218\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") "
Oct 30 17:34:38.923710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.922800    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw\" (UniqueName: \"kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw\") pod \"e396bd00-76e2-4db9-97fc-4d65d04b4218\" (UID: \"e396bd00-76e2-4db9-97fc-4d65d04b4218\") "
Oct 30 17:34:38.923710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.923141    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca" pod "e396bd00-76e2-4db9-97fc-4d65d04b4218" (UID: "e396bd00-76e2-4db9-97fc-4d65d04b4218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:38.924363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.924335    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:38.928744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.928699    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca" pod "e396bd00-76e2-4db9-97fc-4d65d04b4218" (UID: "e396bd00-76e2-4db9-97fc-4d65d04b4218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:38.934972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.934940    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw" pod "e396bd00-76e2-4db9-97fc-4d65d04b4218" (UID: "e396bd00-76e2-4db9-97fc-4d65d04b4218"). InnerVolumeSpecName "kube-api-access-smclw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:38.958946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.958915    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"]
Oct 30 17:34:38.959795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.959773    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e396bd00-76e2-4db9-97fc-4d65d04b4218" containerName="revoke-keys"
Oct 30 17:34:38.960059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.959797    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e396bd00-76e2-4db9-97fc-4d65d04b4218" containerName="revoke-keys"
Oct 30 17:34:38.960059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.960017    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e396bd00-76e2-4db9-97fc-4d65d04b4218" containerName="revoke-keys"
Oct 30 17:34:38.963627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:38.963609    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.013872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.013736    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"]
Oct 30 17:34:39.029301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029257    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029368    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029420    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029479    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029523    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029585    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029618    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029687    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.029916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.029757    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47pd6\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.030047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.030023    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e396bd00-76e2-4db9-97fc-4d65d04b4218-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:39.030131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.030055    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smclw\" (UniqueName: \"kubernetes.io/projected/e396bd00-76e2-4db9-97fc-4d65d04b4218-kube-api-access-smclw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:34:39.035531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.035491    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"]
Oct 30 17:34:39.035721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.035556    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"]
Oct 30 17:34:39.035721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.035699    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.056877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.056839    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:39.131397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131358    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6kj\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.131622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131560    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.131950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131728    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.131950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131775    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.131950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131818    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.131950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131875    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.132184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.131952    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132007    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132112    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.132184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132145    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132173    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.132446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132248    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132298    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.132446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132342    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132376    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132416    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132489    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47pd6\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.132720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.132520    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.133195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.133171    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.133867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.133768    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.134466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.134438    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.134824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.134797    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.136016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.135983    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.137619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.137590    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.139881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.139855    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.164204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.164171    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.164465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.164311    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47pd6\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.233991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.233910    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6kj\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.233991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.233989    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234017    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234041    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234102    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234177    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234212    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234259    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234346    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234397    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234520    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234850    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234914    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.234991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.234943    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.236703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.236678    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.236824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.236807    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.257245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.257217    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6kj\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.257369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.257307    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.301701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.301678    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:39.355886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.355574    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:39.545094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.545011    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:39.630460 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:34:39.629904    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab071bf4_e05f_499b_bb67_9d83c97cd71d.slice/crio-d472e1dccaf2f588cf155ece10e0535d08e76167efe96e4130e22bff2562c48e WatchSource:0}: Error finding container d472e1dccaf2f588cf155ece10e0535d08e76167efe96e4130e22bff2562c48e: Status 404 returned error can't find the container with id d472e1dccaf2f588cf155ece10e0535d08e76167efe96e4130e22bff2562c48e
Oct 30 17:34:39.631182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.630937    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"]
Oct 30 17:34:39.633083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.633025    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:39.633171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.633121    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:39.633171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.633150    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:39.664486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.664441    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"
Oct 30 17:34:39.664742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.664433    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh" event={"ID":"e396bd00-76e2-4db9-97fc-4d65d04b4218","Type":"ContainerDied","Data":"1c1cdabf281b86c1cab0e9f25769880977ff4c4e90abda09814f7de448ef9c6c"}
Oct 30 17:34:39.664742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.664604    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1cdabf281b86c1cab0e9f25769880977ff4c4e90abda09814f7de448ef9c6c"
Oct 30 17:34:39.666498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.666474    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" event={"ID":"ab071bf4-e05f-499b-bb67-9d83c97cd71d","Type":"ContainerStarted","Data":"d472e1dccaf2f588cf155ece10e0535d08e76167efe96e4130e22bff2562c48e"}
Oct 30 17:34:39.712074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.712045    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"]
Oct 30 17:34:39.713379 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:34:39.713354    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9447b2_7531_4460_9c14_4888f084ad65.slice/crio-d60a31dad9c44fe5d3b9c3e457fef7844e0471b27dafb8e99214823c8c3edb5d WatchSource:0}: Error finding container d60a31dad9c44fe5d3b9c3e457fef7844e0471b27dafb8e99214823c8c3edb5d: Status 404 returned error can't find the container with id d60a31dad9c44fe5d3b9c3e457fef7844e0471b27dafb8e99214823c8c3edb5d
Oct 30 17:34:39.716753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.716715    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:39.716875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.716856    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:39.716961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:39.716936    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:34:40.730391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:40.721945    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" event={"ID":"6a9447b2-7531-4460-9c14-4888f084ad65","Type":"ContainerStarted","Data":"a3945924bc7eac34e769d5d42c66ef54f3e3e81cf36db47b21bb1f182e60d7c0"}
Oct 30 17:34:40.730391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:40.722026    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" event={"ID":"6a9447b2-7531-4460-9c14-4888f084ad65","Type":"ContainerStarted","Data":"d60a31dad9c44fe5d3b9c3e457fef7844e0471b27dafb8e99214823c8c3edb5d"}
Oct 30 17:34:40.730391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:40.725861    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:40.730391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:40.727308    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" event={"ID":"ab071bf4-e05f-499b-bb67-9d83c97cd71d","Type":"ContainerStarted","Data":"468b948e172e0ca47d8e72da98e066106e43121e920bc70fb59da3ce67bf9b4f"}
Oct 30 17:34:40.818666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:40.814043    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podStartSLOduration=2.8140168819999998 podStartE2EDuration="2.814016882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:40.788103004 +0000 UTC m=+1286.550146830" watchObservedRunningTime="2026-10-30 17:34:40.814016882 +0000 UTC m=+1286.576060763"
Oct 30 17:34:40.861117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:40.854291    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podStartSLOduration=2.854267888 podStartE2EDuration="2.854267888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:40.848021429 +0000 UTC m=+1286.610065283" watchObservedRunningTime="2026-10-30 17:34:40.854267888 +0000 UTC m=+1286.616311718"
Oct 30 17:34:41.305083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:41.305042    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:41.311858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:41.311819    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 17:34:41.312160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:41.312125    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 17:34:41.371668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:41.362549    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:41.373796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:41.373760    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 17:34:41.374057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:41.374022    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 17:34:42.312150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:42.312113    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 17:34:42.312854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:42.312816    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 17:34:42.357681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:42.356219    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 17:34:42.357681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:42.356275    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 17:34:42.582319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:42.582184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:43.307548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:43.307057    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 17:34:43.307548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:43.307126    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 17:34:43.359398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:43.359362    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 17:34:43.359994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:43.359961    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 17:34:44.103208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.103163    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:44.309893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.309856    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 17:34:44.310191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.310159    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 17:34:44.360726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.357005    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 17:34:44.360726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.357067    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 17:34:44.689720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.689568    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:44.822536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:44.822487    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:45.245681 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:34:45.245618    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:34:45.307146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:45.307105    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 17:34:45.307481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:45.307449    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 17:34:45.366675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:45.362057    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 17:34:45.366675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:45.362132    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 17:34:46.302853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:46.302810    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 17:34:46.303074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:46.302865    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 17:34:46.356027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:46.355991    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 17:34:46.356223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:46.356054    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 17:34:46.702131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:46.702089    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:47.590338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:47.590305    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:47.592598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:47.592573    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:47.596534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:47.596506    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:34:47.626114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:47.626081    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:47.627361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:47.626744    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:47.628390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:47.628362    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:34:48.602547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:48.602507    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:50.344715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:50.344632    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:53.414667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.401736    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:53.414667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.402125    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:34:53.417804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.417763    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:34:53.679842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.679737    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:53.684780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.684741    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:53.697758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.697719    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:53.920668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.917870    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:53.987973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:53.987870    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:34:54.260725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.219156    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:34:54.442696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.425402    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:54.491808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.426768    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:54.669667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.630215    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:54.900519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.767328    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:34:54.900819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.767369    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:34:54.901027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.767390    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:34:54.901146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.767458    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 17:34:54.922050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.920304    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:34:54.991093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:54.991057    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:55.039240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.039213    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:34:55.041350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.041323    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:34:55.041545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.041534    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:34:55.043225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.043211    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:34:55.045087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.045072    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:34:55.047841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.047818    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:34:55.048680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.048667    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:34:55.048980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.048968    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:34:55.049101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.043620    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:34:55.051681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.050388    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:34:55.069112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.069080    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:34:55.113445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.113414    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:34:55.148369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.148333    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:55.307179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.307103    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:34:55.366681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.358811    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:34:55.380087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:55.380059    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:34:56.823198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:56.823165    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:57.795062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:57.794983    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:57.954882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:57.954829    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:58.153022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:58.152982    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:58.854943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:58.854907    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:34:59.681935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.681900    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"]
Oct 30 17:34:59.688455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.688433    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.703760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.703730    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:34:59.709357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.709330    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"]
Oct 30 17:34:59.714463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.714431    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:34:59.760291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.760239    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.760498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.760340    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.760498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.760441    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.861972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.861939    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.862154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.862026    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.862154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.862048    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.862843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.862811    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.865113 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.865087    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:34:59.888228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:34:59.888198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq\") pod \"maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:35:00.010587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:00.010507    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:35:00.323813 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:35:00.323783    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8287d2ab_5a89_4b0b_b06c_d8cc090b2284.slice/crio-c3e995342a3e90b12868185a3368d23dfa1ca30ff1e09da23bd9037566b4611a WatchSource:0}: Error finding container c3e995342a3e90b12868185a3368d23dfa1ca30ff1e09da23bd9037566b4611a: Status 404 returned error can't find the container with id c3e995342a3e90b12868185a3368d23dfa1ca30ff1e09da23bd9037566b4611a
Oct 30 17:35:00.325110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:00.325084    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"]
Oct 30 17:35:00.846488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:00.846449    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:01.000970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:01.000839    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6" event={"ID":"8287d2ab-5a89-4b0b-b06c-d8cc090b2284","Type":"ContainerStarted","Data":"7792d8d63477b1ebb399e1f9114daaebfa6cfd27cbffdceef5908913dd4cab5f"}
Oct 30 17:35:01.000970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:01.000920    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6" event={"ID":"8287d2ab-5a89-4b0b-b06c-d8cc090b2284","Type":"ContainerStarted","Data":"c3e995342a3e90b12868185a3368d23dfa1ca30ff1e09da23bd9037566b4611a"}
Oct 30 17:35:01.047021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:01.046972    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6" podStartSLOduration=2.04695857 podStartE2EDuration="2.04695857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:01.041262687 +0000 UTC m=+1306.803306504" watchObservedRunningTime="2026-10-30 17:35:01.04695857 +0000 UTC m=+1306.809002414"
Oct 30 17:35:02.013337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:02.013289    5534 generic.go:356] "Generic (PLEG): container finished" podID="8287d2ab-5a89-4b0b-b06c-d8cc090b2284" containerID="7792d8d63477b1ebb399e1f9114daaebfa6cfd27cbffdceef5908913dd4cab5f" exitCode=0
Oct 30 17:35:02.013804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:02.013376    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6" event={"ID":"8287d2ab-5a89-4b0b-b06c-d8cc090b2284","Type":"ContainerDied","Data":"7792d8d63477b1ebb399e1f9114daaebfa6cfd27cbffdceef5908913dd4cab5f"}
Oct 30 17:35:02.804283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:02.804241    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:03.197897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.197867    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:35:03.326812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.326784    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca\") pod \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") "
Oct 30 17:35:03.327010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.326923    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca\") pod \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") "
Oct 30 17:35:03.327010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.326961    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq\") pod \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\" (UID: \"8287d2ab-5a89-4b0b-b06c-d8cc090b2284\") "
Oct 30 17:35:03.327226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.327201    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca" pod "8287d2ab-5a89-4b0b-b06c-d8cc090b2284" (UID: "8287d2ab-5a89-4b0b-b06c-d8cc090b2284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:03.327581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.327559    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:03.329051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.329029    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca" pod "8287d2ab-5a89-4b0b-b06c-d8cc090b2284" (UID: "8287d2ab-5a89-4b0b-b06c-d8cc090b2284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:35:03.329185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.329165    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq" pod "8287d2ab-5a89-4b0b-b06c-d8cc090b2284" (UID: "8287d2ab-5a89-4b0b-b06c-d8cc090b2284"). InnerVolumeSpecName "kube-api-access-5z6tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:03.428939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.428867    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:03.428939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:03.428892    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/8287d2ab-5a89-4b0b-b06c-d8cc090b2284-kube-api-access-5z6tq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:04.034162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:04.034127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6" event={"ID":"8287d2ab-5a89-4b0b-b06c-d8cc090b2284","Type":"ContainerDied","Data":"c3e995342a3e90b12868185a3368d23dfa1ca30ff1e09da23bd9037566b4611a"}
Oct 30 17:35:04.034162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:04.034165    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"
Oct 30 17:35:04.034390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:04.034200    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e995342a3e90b12868185a3368d23dfa1ca30ff1e09da23bd9037566b4611a"
Oct 30 17:35:04.504012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:04.503972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:05.086728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:05.086692    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:05.196552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:05.196515    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:05.588393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:05.588355    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:06.088512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:06.088457    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:06.438140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:06.438094    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:06.992129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:06.992091    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:08.235795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:08.235753    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:08.526184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:08.526062    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:09.314318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:09.314283    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:09.902925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:09.902875    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:10.013108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:10.013066    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"]
Oct 30 17:35:10.029959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:10.029926    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"]
Oct 30 17:35:10.185007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:10.184925    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:10.307353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:10.307296    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:10.724842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:10.724788    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:11.460432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:11.460399    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:11.570932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:11.570895    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:11.598877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:11.598775    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"]
Oct 30 17:35:11.623957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:11.623913    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86c97ce8-ls4b6"]
Oct 30 17:35:12.373343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:12.373310    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8287d2ab-5a89-4b0b-b06c-d8cc090b2284" path="/var/lib/kubelet/pods/8287d2ab-5a89-4b0b-b06c-d8cc090b2284/volumes"
Oct 30 17:35:12.724211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:12.724117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:13.466105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:13.466070    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:14.459219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.459181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:14.629280 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.629238    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:14.717872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.717767    5534 scope.go:122] "RemoveContainer" containerID="ad0935354fe95e76bc867cd6f3124dd2862ccd21107bbdd223305452ddd90c64"
Oct 30 17:35:14.730109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.730085    5534 scope.go:122] "RemoveContainer" containerID="b48e95db67e15ff24e968826839a4a892359064a69040f68651441f831858ef3"
Oct 30 17:35:14.741753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.741728    5534 scope.go:122] "RemoveContainer" containerID="1749dabad17deeb3b68769e041611a8d5159d5357c2df483f056e0614d9b2c0f"
Oct 30 17:35:14.752735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.752710    5534 scope.go:122] "RemoveContainer" containerID="6aa3bc994f5771de5b4ffc95b24c2a0765206ab61921e20fca49647ede5d2b25"
Oct 30 17:35:14.765711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.765683    5534 scope.go:122] "RemoveContainer" containerID="3ecd5f2e5ab9d2cee9304330fbd54f9e19298b27f5f890fa1ce53499220902be"
Oct 30 17:35:14.779131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.779101    5534 scope.go:122] "RemoveContainer" containerID="7dfdf68d71c30519595254eb191c40735ff6e1340a1484d2a1d20bdd7fddfedd"
Oct 30 17:35:14.792944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.792919    5534 scope.go:122] "RemoveContainer" containerID="c73e0e8eb33ad64657855f8bbba45849c9946d4a981a6027fc0bd9b4c2fbdd3e"
Oct 30 17:35:14.808324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.808299    5534 scope.go:122] "RemoveContainer" containerID="69e4716895f62ae4285f7fc2b1dfc2b6bcc527e83b9acc375659f74542642814"
Oct 30 17:35:14.819721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.819699    5534 scope.go:122] "RemoveContainer" containerID="07cfee76effcff1d76ea5186836f2cd92b9b8b6fb693d5c92fad014fff028dc0"
Oct 30 17:35:14.830525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.830505    5534 scope.go:122] "RemoveContainer" containerID="eae3a2c14ec7c12eb230597b283c2a1e8457c38743e64522378b5f180f80b0a2"
Oct 30 17:35:14.953369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:14.953333    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:15.014018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.013898    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk" podUID="0d5865a5-c848-48a8-a939-ca628b8f71a8" containerName="payload-processing" containerID="cri-o://3031286eee5003027ffba8ba57cfd5c35806c6452bb1d9a65f2f9ba1f8cdfd5f" gracePeriod=25
Oct 30 17:35:15.030869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.030827    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp" podUID="fdf87969-a5c2-43f1-bbb8-5a837f9b6780" containerName="payload-pre-processing" containerID="cri-o://e43802a23be9573503a34ad281ec681606ccd2064bdc0c750f705c2ede4ad8cf" gracePeriod=25
Oct 30 17:35:15.173110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.173065    5534 generic.go:356] "Generic (PLEG): container finished" podID="0d5865a5-c848-48a8-a939-ca628b8f71a8" containerID="3031286eee5003027ffba8ba57cfd5c35806c6452bb1d9a65f2f9ba1f8cdfd5f" exitCode=0
Oct 30 17:35:15.173505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.173376    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk" event={"ID":"0d5865a5-c848-48a8-a939-ca628b8f71a8","Type":"ContainerDied","Data":"3031286eee5003027ffba8ba57cfd5c35806c6452bb1d9a65f2f9ba1f8cdfd5f"}
Oct 30 17:35:15.176105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.176078    5534 generic.go:356] "Generic (PLEG): container finished" podID="fdf87969-a5c2-43f1-bbb8-5a837f9b6780" containerID="e43802a23be9573503a34ad281ec681606ccd2064bdc0c750f705c2ede4ad8cf" exitCode=0
Oct 30 17:35:15.176468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.176254    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp" event={"ID":"fdf87969-a5c2-43f1-bbb8-5a837f9b6780","Type":"ContainerDied","Data":"e43802a23be9573503a34ad281ec681606ccd2064bdc0c750f705c2ede4ad8cf"}
Oct 30 17:35:15.280598 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:35:15.280469    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:35:15.371904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.371878    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:35:15.376863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.376838    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:35:15.419005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.418976    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume\") pod \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") "
Oct 30 17:35:15.419005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.419012    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr\" (UniqueName: \"kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr\") pod \"0d5865a5-c848-48a8-a939-ca628b8f71a8\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") "
Oct 30 17:35:15.419284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.419089    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume\") pod \"0d5865a5-c848-48a8-a939-ca628b8f71a8\" (UID: \"0d5865a5-c848-48a8-a939-ca628b8f71a8\") "
Oct 30 17:35:15.419284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.419164    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf\" (UniqueName: \"kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf\") pod \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\" (UID: \"fdf87969-a5c2-43f1-bbb8-5a837f9b6780\") "
Oct 30 17:35:15.419405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.419372    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume" pod "fdf87969-a5c2-43f1-bbb8-5a837f9b6780" (UID: "fdf87969-a5c2-43f1-bbb8-5a837f9b6780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:15.419484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.419457    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume" pod "0d5865a5-c848-48a8-a939-ca628b8f71a8" (UID: "0d5865a5-c848-48a8-a939-ca628b8f71a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:15.420269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.420242    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:15.420459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.420437    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5865a5-c848-48a8-a939-ca628b8f71a8-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:15.421764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.421739    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf" pod "fdf87969-a5c2-43f1-bbb8-5a837f9b6780" (UID: "fdf87969-a5c2-43f1-bbb8-5a837f9b6780"). InnerVolumeSpecName "kube-api-access-7n4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:15.421887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.421810    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr" pod "0d5865a5-c848-48a8-a939-ca628b8f71a8" (UID: "0d5865a5-c848-48a8-a939-ca628b8f71a8"). InnerVolumeSpecName "kube-api-access-gq4xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:15.521476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.521440    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4xr\" (UniqueName: \"kubernetes.io/projected/0d5865a5-c848-48a8-a939-ca628b8f71a8-kube-api-access-gq4xr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:15.521476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:15.521472    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4cf\" (UniqueName: \"kubernetes.io/projected/fdf87969-a5c2-43f1-bbb8-5a837f9b6780-kube-api-access-7n4cf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:16.186846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.186806    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk" event={"ID":"0d5865a5-c848-48a8-a939-ca628b8f71a8","Type":"ContainerDied","Data":"218e5afb2a445c6aca7061e661b5a2d54dba64f948beb99143584e4d08391865"}
Oct 30 17:35:16.186846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.186846    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"
Oct 30 17:35:16.187403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.186907    5534 scope.go:122] "RemoveContainer" containerID="3031286eee5003027ffba8ba57cfd5c35806c6452bb1d9a65f2f9ba1f8cdfd5f"
Oct 30 17:35:16.189238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.189220    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"
Oct 30 17:35:16.189394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.189236    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp" event={"ID":"fdf87969-a5c2-43f1-bbb8-5a837f9b6780","Type":"ContainerDied","Data":"44ac631e3c7c7d6845ebce9bad6ceadfc68a0699c8c6564beaee8e9bb85e6d4a"}
Oct 30 17:35:16.201781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.201756    5534 scope.go:122] "RemoveContainer" containerID="e43802a23be9573503a34ad281ec681606ccd2064bdc0c750f705c2ede4ad8cf"
Oct 30 17:35:16.241793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.241771    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"]
Oct 30 17:35:16.253299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.253274    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-86c97ce8-5d44f9cd4-wjspk"]
Oct 30 17:35:16.276543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.276514    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"]
Oct 30 17:35:16.285241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.285169    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-86c97ce8-55c698574c-jc7vp"]
Oct 30 17:35:16.373418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.373385    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5865a5-c848-48a8-a939-ca628b8f71a8" path="/var/lib/kubelet/pods/0d5865a5-c848-48a8-a939-ca628b8f71a8/volumes"
Oct 30 17:35:16.373825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.373811    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf87969-a5c2-43f1-bbb8-5a837f9b6780" path="/var/lib/kubelet/pods/fdf87969-a5c2-43f1-bbb8-5a837f9b6780/volumes"
Oct 30 17:35:16.605006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:16.604976    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:18.395349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:18.395313    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:20.497907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:20.497798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:20.697670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:20.694237    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:20.885670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:20.885455    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:21.640670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:21.635217    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:22.248447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:22.248403    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:35:22.438699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:22.438657    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:35:22.467706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:22.467030    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:23.298669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.291872    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:23.483436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.483400    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:23.781588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.781537    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"]
Oct 30 17:35:23.782629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.782594    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fdf87969-a5c2-43f1-bbb8-5a837f9b6780" containerName="payload-pre-processing"
Oct 30 17:35:23.782858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.782839    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf87969-a5c2-43f1-bbb8-5a837f9b6780" containerName="payload-pre-processing"
Oct 30 17:35:23.782985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.782973    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8287d2ab-5a89-4b0b-b06c-d8cc090b2284" containerName="revoke-keys"
Oct 30 17:35:23.783070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.783059    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8287d2ab-5a89-4b0b-b06c-d8cc090b2284" containerName="revoke-keys"
Oct 30 17:35:23.783157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.783147    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0d5865a5-c848-48a8-a939-ca628b8f71a8" containerName="payload-processing"
Oct 30 17:35:23.783233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.783223    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5865a5-c848-48a8-a939-ca628b8f71a8" containerName="payload-processing"
Oct 30 17:35:23.783518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.783503    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf87969-a5c2-43f1-bbb8-5a837f9b6780" containerName="payload-pre-processing"
Oct 30 17:35:23.783630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.783618    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8287d2ab-5a89-4b0b-b06c-d8cc090b2284" containerName="revoke-keys"
Oct 30 17:35:23.783735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.783724    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5865a5-c848-48a8-a939-ca628b8f71a8" containerName="payload-processing"
Oct 30 17:35:23.802211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.796549    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:23.809622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.809586    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"]
Oct 30 17:35:23.838077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.826173    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:23.838077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.832033    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a17354\""
Oct 30 17:35:23.838077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.832173    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a17354-dockercfg-lqmlg\""
Oct 30 17:35:23.881367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.881321    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tkk\" (UniqueName: \"kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk\") pod \"payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:23.892309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:23.881676    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:24.000095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.000057    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tkk\" (UniqueName: \"kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk\") pod \"payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:24.000968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.000933    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:24.001863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.001838    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:24.025403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.025366    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tkk\" (UniqueName: \"kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk\") pod \"payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:24.183175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.183135    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:24.450385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.450303    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:24.540843 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:35:24.540811    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7395335_85f2_4166_a73f_f29c5fa1e026.slice/crio-2573125da6fff2770733f50a0fcd8edacf38cef157ddacd10627de1f773b76d1 WatchSource:0}: Error finding container 2573125da6fff2770733f50a0fcd8edacf38cef157ddacd10627de1f773b76d1: Status 404 returned error can't find the container with id 2573125da6fff2770733f50a0fcd8edacf38cef157ddacd10627de1f773b76d1
Oct 30 17:35:24.543143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.543118    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"]
Oct 30 17:35:24.657041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.657006    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:24.995942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:24.995907    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:25.063308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.063268    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"]
Oct 30 17:35:25.075035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.074995    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed5fc69-9m8p4"]
Oct 30 17:35:25.401926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.401886    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm" event={"ID":"e7395335-85f2-4166-a73f-f29c5fa1e026","Type":"ContainerStarted","Data":"f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f"}
Oct 30 17:35:25.402140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.401954    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm" event={"ID":"e7395335-85f2-4166-a73f-f29c5fa1e026","Type":"ContainerStarted","Data":"2573125da6fff2770733f50a0fcd8edacf38cef157ddacd10627de1f773b76d1"}
Oct 30 17:35:25.402265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.402248    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:25.452275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.452227    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm" podStartSLOduration=2.452214882 podStartE2EDuration="2.452214882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:25.451122929 +0000 UTC m=+1331.213166744" watchObservedRunningTime="2026-10-30 17:35:25.452214882 +0000 UTC m=+1331.214258694"
Oct 30 17:35:25.606516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.606456    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:25.745281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.745194    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:25.957668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:25.940727    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:26.371194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:26.371093    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38b74e4-74b9-49e3-a58c-15d004db0cb2" path="/var/lib/kubelet/pods/e38b74e4-74b9-49e3-a58c-15d004db0cb2/volumes"
Oct 30 17:35:27.707746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:27.707711    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:28.156239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:28.156204    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:29.427789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:29.427753    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:29.721809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:29.721697    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:31.123801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.123774    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:31.288187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.288151    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"]
Oct 30 17:35:31.294089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.294069    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.312488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.312460    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:35:31.312653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.312548    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:35:31.331453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.331425    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"]
Oct 30 17:35:31.443210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.443109    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.443397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.443338    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.443481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.443444    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kgf\" (UniqueName: \"kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.544394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.544362    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kgf\" (UniqueName: \"kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.544564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.544482    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.544634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.544583    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.545350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.545330    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.547005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.546983    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.567303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.567276    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kgf\" (UniqueName: \"kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf\") pod \"maas-api-revoke-sub-22c10c48-lncgn\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.609428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.609401    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:31.921191 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:35:31.921147    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19758f36_3669_41d9_ac48_5cac16aabfb0.slice/crio-b887d1b4d6abafb78cc4c3ff0d2af4fbcbf5871b94547c0afe31c05277aaf3a2 WatchSource:0}: Error finding container b887d1b4d6abafb78cc4c3ff0d2af4fbcbf5871b94547c0afe31c05277aaf3a2: Status 404 returned error can't find the container with id b887d1b4d6abafb78cc4c3ff0d2af4fbcbf5871b94547c0afe31c05277aaf3a2
Oct 30 17:35:31.923051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:31.923027    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"]
Oct 30 17:35:32.472302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:32.472255    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn" event={"ID":"19758f36-3669-41d9-ac48-5cac16aabfb0","Type":"ContainerStarted","Data":"3dfc11cd0c154f3dcf434e18ce1cedc2ba0f97ca1b643c71469b29909305b4c6"}
Oct 30 17:35:32.472776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:32.472333    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn" event={"ID":"19758f36-3669-41d9-ac48-5cac16aabfb0","Type":"ContainerStarted","Data":"b887d1b4d6abafb78cc4c3ff0d2af4fbcbf5871b94547c0afe31c05277aaf3a2"}
Oct 30 17:35:32.518308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:32.518192    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn" podStartSLOduration=1.5181733039999998 podStartE2EDuration="1.518173304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:32.514922948 +0000 UTC m=+1338.276966761" watchObservedRunningTime="2026-10-30 17:35:32.518173304 +0000 UTC m=+1338.280217119"
Oct 30 17:35:32.927153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:32.927119    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:33.482381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:33.482345    5534 generic.go:356] "Generic (PLEG): container finished" podID="19758f36-3669-41d9-ac48-5cac16aabfb0" containerID="3dfc11cd0c154f3dcf434e18ce1cedc2ba0f97ca1b643c71469b29909305b4c6" exitCode=0
Oct 30 17:35:33.482861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:33.482424    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn" event={"ID":"19758f36-3669-41d9-ac48-5cac16aabfb0","Type":"ContainerDied","Data":"3dfc11cd0c154f3dcf434e18ce1cedc2ba0f97ca1b643c71469b29909305b4c6"}
Oct 30 17:35:34.662824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.662799    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:34.720710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.720673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:34.794499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.794401    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf\" (UniqueName: \"kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf\") pod \"19758f36-3669-41d9-ac48-5cac16aabfb0\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") "
Oct 30 17:35:34.795377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.795337    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca\") pod \"19758f36-3669-41d9-ac48-5cac16aabfb0\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") "
Oct 30 17:35:34.795532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.795407    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca\") pod \"19758f36-3669-41d9-ac48-5cac16aabfb0\" (UID: \"19758f36-3669-41d9-ac48-5cac16aabfb0\") "
Oct 30 17:35:34.796660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.796540    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca" pod "19758f36-3669-41d9-ac48-5cac16aabfb0" (UID: "19758f36-3669-41d9-ac48-5cac16aabfb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:34.797192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.797167    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:34.798457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.798413    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca" pod "19758f36-3669-41d9-ac48-5cac16aabfb0" (UID: "19758f36-3669-41d9-ac48-5cac16aabfb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:35:34.807316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.807261    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf" pod "19758f36-3669-41d9-ac48-5cac16aabfb0" (UID: "19758f36-3669-41d9-ac48-5cac16aabfb0"). InnerVolumeSpecName "kube-api-access-x2kgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:34.899038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.899002    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kgf\" (UniqueName: \"kubernetes.io/projected/19758f36-3669-41d9-ac48-5cac16aabfb0-kube-api-access-x2kgf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:34.899038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:34.899036    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19758f36-3669-41d9-ac48-5cac16aabfb0-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:35.512927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:35.512888    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"
Oct 30 17:35:35.513149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:35.512887    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn" event={"ID":"19758f36-3669-41d9-ac48-5cac16aabfb0","Type":"ContainerDied","Data":"b887d1b4d6abafb78cc4c3ff0d2af4fbcbf5871b94547c0afe31c05277aaf3a2"}
Oct 30 17:35:35.513149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:35.513094    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b887d1b4d6abafb78cc4c3ff0d2af4fbcbf5871b94547c0afe31c05277aaf3a2"
Oct 30 17:35:36.274670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.274613    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"]
Oct 30 17:35:36.275342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.275328    5534 cpu_manager.go:414] "containerMap: removing container" podUID="19758f36-3669-41d9-ac48-5cac16aabfb0" containerName="revoke-keys"
Oct 30 17:35:36.275395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.275343    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19758f36-3669-41d9-ac48-5cac16aabfb0" containerName="revoke-keys"
Oct 30 17:35:36.275497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.275484    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="19758f36-3669-41d9-ac48-5cac16aabfb0" containerName="revoke-keys"
Oct 30 17:35:36.280969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.280946    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.298037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.298011    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a17354-serving-cert\""
Oct 30 17:35:36.317117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.317082    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"]
Oct 30 17:35:36.335101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.335059    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbr9\" (UniqueName: \"kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.335298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.335261    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.335378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.335330    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.338662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.335457    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.374978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.374926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:36.416499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.416472    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 17:35:36.436917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.436874    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbr9\" (UniqueName: \"kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.437801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.437154    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.437801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.437244    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.437801 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:35:36.437275    5534 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-serving-cert: secret "maas-api-e2e-shared-a-w2-a17354-serving-cert" not found
Oct 30 17:35:36.437801 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:35:36.437333    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls podName:639be5d9-7161-485f-9d31-d5f0714a9f21 nodeName:}" failed. No retries permitted until 2026-10-30 17:35:36.937315222 +0000 UTC m=+1342.699359030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls") pod "maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" (UID: "639be5d9-7161-485f-9d31-d5f0714a9f21") : secret "maas-api-e2e-shared-a-w2-a17354-serving-cert" not found
Oct 30 17:35:36.437801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.437765    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.438474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.438447    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.440141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.440120    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.463265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.463231    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbr9\" (UniqueName: \"kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.508064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.508013    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:36.705358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.705321    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:36.973025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.972935    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:36.976170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:36.976142    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:37.217798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:37.217769    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:37.567519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:37.567492    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"]
Oct 30 17:35:37.569323 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:35:37.569294    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639be5d9_7161_485f_9d31_d5f0714a9f21.slice/crio-3c4fc4729888a6f33d620f62fa2ddc257b9a615d78527497a42c3e50fb7e1445 WatchSource:0}: Error finding container 3c4fc4729888a6f33d620f62fa2ddc257b9a615d78527497a42c3e50fb7e1445: Status 404 returned error can't find the container with id 3c4fc4729888a6f33d620f62fa2ddc257b9a615d78527497a42c3e50fb7e1445
Oct 30 17:35:38.460557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:38.460519    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:38.585516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:38.585476    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" event={"ID":"639be5d9-7161-485f-9d31-d5f0714a9f21","Type":"ContainerStarted","Data":"2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97"}
Oct 30 17:35:38.585967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:38.585543    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" event={"ID":"639be5d9-7161-485f-9d31-d5f0714a9f21","Type":"ContainerStarted","Data":"3c4fc4729888a6f33d620f62fa2ddc257b9a615d78527497a42c3e50fb7e1445"}
Oct 30 17:35:38.585967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:38.585576    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:38.642723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:38.642690    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:38.651995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:38.651931    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" podStartSLOduration=2.245313478 podStartE2EDuration="2.651912121s" totalImagesPullingTime="406.598643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:38.642787138 +0000 UTC m=+1344.404830952" watchObservedRunningTime="2026-10-30 17:35:38.651912121 +0000 UTC m=+1344.413955934"
Oct 30 17:35:39.680609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:39.680578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:40.182520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.182486    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:40.338680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.338626    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:40.407663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.406506    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"]
Oct 30 17:35:40.412365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.412337    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:40.435182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.435121    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"]
Oct 30 17:35:40.442282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.442248    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81e9336e-dockercfg-h9b4n\""
Oct 30 17:35:40.442428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.442323    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81e9336e\""
Oct 30 17:35:40.507186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.507160    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:40.551947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.551903    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:40.552149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.551965    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdft\" (UniqueName: \"kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:40.678665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.671772    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:40.678665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.671825    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdft\" (UniqueName: \"kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:40.678665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.673065    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:40.736821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:40.735795    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdft\" (UniqueName: \"kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:41.028079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.027989    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:41.425219 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:35:41.425188    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777dd8b3_39cb_4ef2_9980_fccaa02d3650.slice/crio-0a4035bb9fd6a3ac057e7f18fb10ba764c708f43873c7526a8efed2c2a1ae402 WatchSource:0}: Error finding container 0a4035bb9fd6a3ac057e7f18fb10ba764c708f43873c7526a8efed2c2a1ae402: Status 404 returned error can't find the container with id 0a4035bb9fd6a3ac057e7f18fb10ba764c708f43873c7526a8efed2c2a1ae402
Oct 30 17:35:41.427269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.427242    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"]
Oct 30 17:35:41.765392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.765303    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"]
Oct 30 17:35:41.778837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.778802    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" event={"ID":"777dd8b3-39cb-4ef2-9980-fccaa02d3650","Type":"ContainerStarted","Data":"7ed2bf3149cd0030eaea5380aed6f5bf036554b09468c6cf79dbcad4ca9112b0"}
Oct 30 17:35:41.779072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.779057    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:41.779184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.779171    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" event={"ID":"777dd8b3-39cb-4ef2-9980-fccaa02d3650","Type":"ContainerStarted","Data":"0a4035bb9fd6a3ac057e7f18fb10ba764c708f43873c7526a8efed2c2a1ae402"}
Oct 30 17:35:41.779364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.779350    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.782529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.782503    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"]
Oct 30 17:35:41.797325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.794870    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:35:41.799208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.799187    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:35:41.815050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.814993    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" podStartSLOduration=1.814975451 podStartE2EDuration="1.814975451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:41.812847553 +0000 UTC m=+1347.574891366" watchObservedRunningTime="2026-10-30 17:35:41.814975451 +0000 UTC m=+1347.577019266"
Oct 30 17:35:41.890946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.890896    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.891152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.891091    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqg7d\" (UniqueName: \"kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.891152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.891139    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.992514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.992476    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.992761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.992601    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqg7d\" (UniqueName: \"kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.992761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.992657    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.993506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.993482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:41.997222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:41.997201    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:42.025919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.025849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqg7d\" (UniqueName: \"kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d\") pod \"maas-api-revoke-sub-cc985012-rxd8k\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:42.102064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.102026    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:42.292936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.292840    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:42.421076 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:35:42.421048    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e817252_4924_4ab5_8bda_5c39661a1105.slice/crio-c1048712a6801383e752f7e74c7f8904218d9b511ca8dbeed57e92f12aedb0e8 WatchSource:0}: Error finding container c1048712a6801383e752f7e74c7f8904218d9b511ca8dbeed57e92f12aedb0e8: Status 404 returned error can't find the container with id c1048712a6801383e752f7e74c7f8904218d9b511ca8dbeed57e92f12aedb0e8
Oct 30 17:35:42.422631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.422598    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"]
Oct 30 17:35:42.449528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.449489    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:42.618717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.612297    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:42.784667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.782794    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k" event={"ID":"9e817252-4924-4ab5-8bda-5c39661a1105","Type":"ContainerStarted","Data":"c1048712a6801383e752f7e74c7f8904218d9b511ca8dbeed57e92f12aedb0e8"}
Oct 30 17:35:42.946167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:42.946079    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:43.798303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:43.798202    5534 generic.go:356] "Generic (PLEG): container finished" podID="9e817252-4924-4ab5-8bda-5c39661a1105" containerID="f2fded00bcf3fd74c52edab7368c1bd22c9f685a4e8f3c5075530d519947b5b2" exitCode=0
Oct 30 17:35:43.798809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:43.798300    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k" event={"ID":"9e817252-4924-4ab5-8bda-5c39661a1105","Type":"ContainerDied","Data":"f2fded00bcf3fd74c52edab7368c1bd22c9f685a4e8f3c5075530d519947b5b2"}
Oct 30 17:35:44.601046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:44.601013    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 17:35:44.767484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:44.767450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:45.024794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.024755    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:45.056377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.056351    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca\") pod \"9e817252-4924-4ab5-8bda-5c39661a1105\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") "
Oct 30 17:35:45.056521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.056468    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d\" (UniqueName: \"kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d\") pod \"9e817252-4924-4ab5-8bda-5c39661a1105\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") "
Oct 30 17:35:45.056597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.056563    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca\") pod \"9e817252-4924-4ab5-8bda-5c39661a1105\" (UID: \"9e817252-4924-4ab5-8bda-5c39661a1105\") "
Oct 30 17:35:45.057109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.057082    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca" pod "9e817252-4924-4ab5-8bda-5c39661a1105" (UID: "9e817252-4924-4ab5-8bda-5c39661a1105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:45.057339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.057318    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:45.059342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.059315    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca" pod "9e817252-4924-4ab5-8bda-5c39661a1105" (UID: "9e817252-4924-4ab5-8bda-5c39661a1105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:35:45.059803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.059782    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d" pod "9e817252-4924-4ab5-8bda-5c39661a1105" (UID: "9e817252-4924-4ab5-8bda-5c39661a1105"). InnerVolumeSpecName "kube-api-access-qqg7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:45.159504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.159425    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e817252-4924-4ab5-8bda-5c39661a1105-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:45.159504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.159454    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqg7d\" (UniqueName: \"kubernetes.io/projected/9e817252-4924-4ab5-8bda-5c39661a1105-kube-api-access-qqg7d\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:35:45.244561 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:35:45.244519    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:35:45.821454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.821428    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"
Oct 30 17:35:45.821667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.821418    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k" event={"ID":"9e817252-4924-4ab5-8bda-5c39661a1105","Type":"ContainerDied","Data":"c1048712a6801383e752f7e74c7f8904218d9b511ca8dbeed57e92f12aedb0e8"}
Oct 30 17:35:45.821667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:45.821574    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1048712a6801383e752f7e74c7f8904218d9b511ca8dbeed57e92f12aedb0e8"
Oct 30 17:35:46.845902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:46.845872    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:48.890860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:48.890822    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:50.760493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:50.757423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:52.054658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.054613    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"]
Oct 30 17:35:52.071348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.071321    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9cb06a-lx44h"]
Oct 30 17:35:52.375393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.375350    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b0f57b-e484-4e3c-8e16-c552dcdc0c9b" path="/var/lib/kubelet/pods/21b0f57b-e484-4e3c-8e16-c552dcdc0c9b/volumes"
Oct 30 17:35:52.669093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.668950    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:52.786453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.786428    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:35:52.956523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.956446    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"]
Oct 30 17:35:52.957586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.957566    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9e817252-4924-4ab5-8bda-5c39661a1105" containerName="revoke-keys"
Oct 30 17:35:52.957670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.957591    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e817252-4924-4ab5-8bda-5c39661a1105" containerName="revoke-keys"
Oct 30 17:35:52.957811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.957798    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e817252-4924-4ab5-8bda-5c39661a1105" containerName="revoke-keys"
Oct 30 17:35:52.975736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.975701    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:52.994865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.994834    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7bb2e69d-openshift-default-dockercfg-pn9dq\""
Oct 30 17:35:52.999873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:52.999850    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"]
Oct 30 17:35:53.024538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.024499    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:53.096814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.096761    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.096899    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097016    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7wn\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097063    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097094    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097145    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097416    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097562    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.097794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.097636    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.200193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.200145    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.200620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.200481    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.201057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.201024    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.201239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.201211    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.201458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.201443    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7wn\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.201632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.201606    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.201799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.201785    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.204798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.204673    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.204798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.204728    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.204996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.204862    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.205634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.205612    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.213738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.206403    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.213738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.207134    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.213738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.207458    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.226801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.224471    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.226801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.225795    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.270666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.265392    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.270666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.266198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7wn\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn\") pod \"e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") " pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.270666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.270500    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:53.303883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.303852    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:53.664146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.664107    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:35:53.664244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.664198    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:35:53.664290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.664245    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:35:53.665046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.665020    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"]
Oct 30 17:35:53.916677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.916554    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" event={"ID":"2f7cc5c1-f70f-4fd6-a347-5b40db00e160","Type":"ContainerStarted","Data":"b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"}
Oct 30 17:35:53.916677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.916616    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" event={"ID":"2f7cc5c1-f70f-4fd6-a347-5b40db00e160","Type":"ContainerStarted","Data":"f9cbbfe8a5e1431bc909cf4df7fe48d4aff71d16182afd7585ffedb948122c97"}
Oct 30 17:35:53.969778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:53.969616    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podStartSLOduration=1.969590568 podStartE2EDuration="1.969590568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:53.962109474 +0000 UTC m=+1359.724153284" watchObservedRunningTime="2026-10-30 17:35:53.969590568 +0000 UTC m=+1359.731634382"
Oct 30 17:35:54.217823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.217729    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:54.304443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.304404    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:54.310687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.310630    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:54.377573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.377537    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:54.521718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.521575    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:54.674599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.674564    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:54.925468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.925441    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:54.926527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:54.926506    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:35:55.087665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:55.079489    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:55.545931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:55.545899    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:55.786529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:55.786494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:55.975509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:55.975473    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:56.679011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.678976    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"]
Oct 30 17:35:56.683926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.683903    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.736032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.735994    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-81e9336e-serving-cert\""
Oct 30 17:35:56.743686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.743612    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"]
Oct 30 17:35:56.780376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.780341    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.780624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.780403    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.780624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.780558    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.780624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.780583    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px95l\" (UniqueName: \"kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.835762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.835727    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:56.883676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.882173    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.883676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.882258    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.883676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.882344    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.883676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.882375    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px95l\" (UniqueName: \"kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.883676 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:35:56.882856    5534 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-serving-cert: secret "maas-api-e2e-mt-81e9336e-serving-cert" not found
Oct 30 17:35:56.883676 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:35:56.882948    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls podName:a777fcc3-983e-4a83-9de3-29dc81ec1cc2 nodeName:}" failed. No retries permitted until 2026-10-30 17:35:57.382925641 +0000 UTC m=+1363.144969451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls") pod "maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" (UID: "a777fcc3-983e-4a83-9de3-29dc81ec1cc2") : secret "maas-api-e2e-mt-81e9336e-serving-cert" not found
Oct 30 17:35:56.886829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.886798    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.927670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.919209    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:56.932077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:56.931357    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px95l\" (UniqueName: \"kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:57.107033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:57.106998    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:57.328815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:57.328733    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:57.390741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:57.390704    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:57.393453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:57.393423    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:57.606284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:57.606254    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:57.944558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:57.944529    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"]
Oct 30 17:35:58.972949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:58.972907    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" event={"ID":"a777fcc3-983e-4a83-9de3-29dc81ec1cc2","Type":"ContainerStarted","Data":"023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a"}
Oct 30 17:35:58.973464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:58.972967    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" event={"ID":"a777fcc3-983e-4a83-9de3-29dc81ec1cc2","Type":"ContainerStarted","Data":"882fe726f12981e9fb7b88f77f814d5f2aefb821181a7d0f6bd3e3ea64cf37e1"}
Oct 30 17:35:58.973464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:58.973054    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:35:59.026052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.026004    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" podStartSLOduration=2.70144202 podStartE2EDuration="3.025989575s" totalImagesPullingTime="324.547555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:59.024151345 +0000 UTC m=+1364.786195159" watchObservedRunningTime="2026-10-30 17:35:59.025989575 +0000 UTC m=+1364.788033387"
Oct 30 17:35:59.722937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.722861    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:59.741856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.741823    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"]
Oct 30 17:35:59.748009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.747977    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:35:59.764899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.764873    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7bb2e69d\""
Oct 30 17:35:59.784082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.784056    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"]
Oct 30 17:35:59.835794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.835756    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txv68\" (UniqueName: \"kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68\") pod \"payload-processing-e2e-mt-7bb2e69d-68d847746-llq59\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") " pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:35:59.836177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.836152    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume\") pod \"payload-processing-e2e-mt-7bb2e69d-68d847746-llq59\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") " pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:35:59.974459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.939760    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume\") pod \"payload-processing-e2e-mt-7bb2e69d-68d847746-llq59\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") " pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:35:59.974459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.939905    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txv68\" (UniqueName: \"kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68\") pod \"payload-processing-e2e-mt-7bb2e69d-68d847746-llq59\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") " pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:35:59.974459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.966116    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:35:59.974459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.972266    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume\") pod \"payload-processing-e2e-mt-7bb2e69d-68d847746-llq59\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") " pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:35:59.981470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:35:59.981230    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txv68\" (UniqueName: \"kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68\") pod \"payload-processing-e2e-mt-7bb2e69d-68d847746-llq59\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") " pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:36:00.065675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:00.064954    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:36:00.402215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:00.402184    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"]
Oct 30 17:36:00.411293 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:36:00.411253    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd488f290_f98c_42b7_b0f5_572d1dc96d34.slice/crio-18fcbbd3d6dc36bec876ec71648334a3ea7e24351e3feea7c4bd16ee72414d5b WatchSource:0}: Error finding container 18fcbbd3d6dc36bec876ec71648334a3ea7e24351e3feea7c4bd16ee72414d5b: Status 404 returned error can't find the container with id 18fcbbd3d6dc36bec876ec71648334a3ea7e24351e3feea7c4bd16ee72414d5b
Oct 30 17:36:01.022739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:01.022699    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59" event={"ID":"d488f290-f98c-42b7-b0f5-572d1dc96d34","Type":"ContainerStarted","Data":"2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248"}
Oct 30 17:36:01.023184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:01.022766    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59" event={"ID":"d488f290-f98c-42b7-b0f5-572d1dc96d34","Type":"ContainerStarted","Data":"18fcbbd3d6dc36bec876ec71648334a3ea7e24351e3feea7c4bd16ee72414d5b"}
Oct 30 17:36:01.023184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:01.022836    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:36:01.069481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:01.068730    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59" podStartSLOduration=2.068711504 podStartE2EDuration="2.068711504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:01.066854463 +0000 UTC m=+1366.828898344" watchObservedRunningTime="2026-10-30 17:36:01.068711504 +0000 UTC m=+1366.830755317"
Oct 30 17:36:02.484853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.484806    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"]
Oct 30 17:36:02.491075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.491045    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.514750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.514715    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-016f1d-openshift-default-dockercfg-jvhbm\""
Oct 30 17:36:02.547578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.547536    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"]
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676253    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676435    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676483    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676552    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676662    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7btm\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676732    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676794    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676945    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.676987    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.686717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.678159    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787017    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787080    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787151    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787290    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787326    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787388    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787469    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7btm\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787531    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.787588    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.788241    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.807729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.789026    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.808924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.808891    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.819739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.809682    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.819985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.819771    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.820108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.820036    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.840431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.840395    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.846352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.846317    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7btm\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.849171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.849127    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:02.905225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:02.904923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:03.111696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:03.111656    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:03.483004 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:36:03.482967    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc55c6d_f8ec_405c_82b3_6611f2ca26c1.slice/crio-334d3e25fb4f396d6b7ed71897cdbed528b2d3ac25f322f9a9f4bee116af7c57 WatchSource:0}: Error finding container 334d3e25fb4f396d6b7ed71897cdbed528b2d3ac25f322f9a9f4bee116af7c57: Status 404 returned error can't find the container with id 334d3e25fb4f396d6b7ed71897cdbed528b2d3ac25f322f9a9f4bee116af7c57
Oct 30 17:36:03.484373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:03.484347    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"]
Oct 30 17:36:03.486224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:03.486191    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:36:03.486519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:03.486263    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:36:03.486519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:03.486293    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:36:03.665734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:03.659048    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:04.076012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:04.075963    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" event={"ID":"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1","Type":"ContainerStarted","Data":"db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"}
Oct 30 17:36:04.076224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:04.076033    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" event={"ID":"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1","Type":"ContainerStarted","Data":"334d3e25fb4f396d6b7ed71897cdbed528b2d3ac25f322f9a9f4bee116af7c57"}
Oct 30 17:36:04.112777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:04.112729    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:04.119459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:04.119428    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:04.174868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:04.174813    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podStartSLOduration=2.174796162 podStartE2EDuration="2.174796162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:04.16691274 +0000 UTC m=+1369.928956552" watchObservedRunningTime="2026-10-30 17:36:04.174796162 +0000 UTC m=+1369.936839974"
Oct 30 17:36:04.438382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:04.438336    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:05.017397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:05.017372    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:36:05.091018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:05.090987    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:05.092170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:05.092145    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:36:06.345102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:06.345065    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:06.835906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:06.835824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:07.022633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:07.022594    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:08.325948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:08.325911    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:08.552317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:08.552287    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:08.900247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:08.900212    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:09.117666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:09.111584    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:11.039361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:11.039324    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"]
Oct 30 17:36:11.050678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:11.050636    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04764903-ztv6b"]
Oct 30 17:36:11.189201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:11.188898    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:12.045727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:12.045700    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:36:12.373206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:12.373169    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f0f0bb-d14a-40eb-ae2f-d9b059c4f496" path="/var/lib/kubelet/pods/30f0f0bb-d14a-40eb-ae2f-d9b059c4f496/volumes"
Oct 30 17:36:12.986029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:12.985993    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:13.186021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:13.185988    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:13.404263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:13.404231    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:14.069213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.069175    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"]
Oct 30 17:36:14.082869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.082839    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd98076-vgwb8"]
Oct 30 17:36:14.376066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.376034    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b6b3d1-ce85-479f-9f10-5f7e767ff6de" path="/var/lib/kubelet/pods/c0b6b3d1-ce85-479f-9f10-5f7e767ff6de/volumes"
Oct 30 17:36:14.855672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.855625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:14.897121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.897088    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"]
Oct 30 17:36:14.903804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.903776    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:14.941805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.941780    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-8d1062\""
Oct 30 17:36:14.948431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.948400    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddfb\" (UniqueName: \"kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb\") pod \"payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:14.948571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.948515    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:14.951849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.951823    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"]
Oct 30 17:36:14.976605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.976569    5534 scope.go:122] "RemoveContainer" containerID="4c4f4e419ef484e9ea24c285796d75ad26a35cf7997c752e97d9e23a8bd1b89f"
Oct 30 17:36:14.991397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:14.991372    5534 scope.go:122] "RemoveContainer" containerID="587e7b0cb22b968268ec6a199c9829c4e44876f163999fc531ef0835082e6adb"
Oct 30 17:36:15.057342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.057165    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddfb\" (UniqueName: \"kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb\") pod \"payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:15.059072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.057606    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:15.059072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.058745    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:15.159756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.149370    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddfb\" (UniqueName: \"kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb\") pod \"payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:15.210190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.209927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:15.230160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.220906    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:15.332500 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:36:15.332456    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:36:15.683853 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:36:15.683819    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2a645e_dbe2_4182_beff_2b3e9d59bab3.slice/crio-6561e121d6f7b3a086a9694bdd2a519cc3c67275172d1d6821bb714519f4379d WatchSource:0}: Error finding container 6561e121d6f7b3a086a9694bdd2a519cc3c67275172d1d6821bb714519f4379d: Status 404 returned error can't find the container with id 6561e121d6f7b3a086a9694bdd2a519cc3c67275172d1d6821bb714519f4379d
Oct 30 17:36:15.684855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.684827    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"]
Oct 30 17:36:15.899747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:15.886217    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:16.352591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:16.352556    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" event={"ID":"cc2a645e-dbe2-4182-beff-2b3e9d59bab3","Type":"ContainerStarted","Data":"040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98"}
Oct 30 17:36:16.352836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:16.352631    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" event={"ID":"cc2a645e-dbe2-4182-beff-2b3e9d59bab3","Type":"ContainerStarted","Data":"6561e121d6f7b3a086a9694bdd2a519cc3c67275172d1d6821bb714519f4379d"}
Oct 30 17:36:16.352836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:16.352821    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:16.442710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:16.442615    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" podStartSLOduration=2.442598873 podStartE2EDuration="2.442598873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:16.432858514 +0000 UTC m=+1382.194902340" watchObservedRunningTime="2026-10-30 17:36:16.442598873 +0000 UTC m=+1382.204642686"
Oct 30 17:36:17.777184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:17.777145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:19.677675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:19.676779    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:19.999164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:19.999068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:20.052595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:20.052569    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"]
Oct 30 17:36:20.059627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:20.059596    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651e6b40-hhrqz"]
Oct 30 17:36:20.303043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:20.302941    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:20.448668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:20.441535    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e3b990-44d6-41a0-ad6a-03531d2a6327" path="/var/lib/kubelet/pods/d3e3b990-44d6-41a0-ad6a-03531d2a6327/volumes"
Oct 30 17:36:21.482861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:21.465936    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 17:36:21.616800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:21.616752    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.042751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.042716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:22.072164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.072081    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.119541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.119498    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.208302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.208263    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:36:22.322890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.322780    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.363871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.363811    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.495670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.475929    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.542797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.542728    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.569144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.560719    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.589909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.589883    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:36:22.633909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.632787    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:36:22.633909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.633548    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:36:22.733227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.733185    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:22.873720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:22.854786    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:23.076961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.076915    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:23.077791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.077770    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:36:23.129821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.114658    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:36:23.208257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.208123    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:23.347666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.345037    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:23.347666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.345556    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:36:23.362683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.362613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:23.379544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.379487    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:23.399886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.380420    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:36:23.523748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.523715    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"]
Oct 30 17:36:23.546803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.531825    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.549591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.549562    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.576583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.576550    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.576985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.576959    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xdm\" (UniqueName: \"kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.577550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.577524    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.577898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.573304    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:23.602587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.602557    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7bb2e69d-serving-cert\""
Oct 30 17:36:23.647668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.645516    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.688618    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.688738    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.688801    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xdm\" (UniqueName: \"kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.689412    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.689974    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:36:23.691129    5534 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-serving-cert: secret "maas-api-e2e-mt-7bb2e69d-serving-cert" not found
Oct 30 17:36:23.691718 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:36:23.691190    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls podName:d08d4ddd-ba79-4fe3-899a-619e7b921548 nodeName:}" failed. No retries permitted until 2026-10-30 17:36:24.191170463 +0000 UTC m=+1389.953214271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls") pod "maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" (UID: "d08d4ddd-ba79-4fe3-899a-619e7b921548") : secret "maas-api-e2e-mt-7bb2e69d-serving-cert" not found
Oct 30 17:36:23.714674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.705831    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.740667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.737172    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:36:23.740667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.737558    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:36:23.770671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.759512    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:23.865584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.865552    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"]
Oct 30 17:36:23.905364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:23.905332    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xdm\" (UniqueName: \"kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:24.084776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.080880    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:36:24.138174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.138132    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:36:24.181748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.138439    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:24.241962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.232729    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:36:24.317667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.297395    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:24.317667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.298363    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:24.354913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.351729    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:24.359687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.356298    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\") pod \"maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:24.402784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.395111    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:24.450665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.441486    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:24.450665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.442008    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:36:24.492935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.490850    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:24.565928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.565896    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:24.667706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.666373    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:36:24.700475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.700431    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:24.732227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.726455    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:36:24.741764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.741730    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 17:36:24.831788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.831407    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:24.888664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.886284    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:36:24.911733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.888981    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:36:24.913043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.913018    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:36:24.942738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:24.942704    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:36:25.212711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:25.210674    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"]
Oct 30 17:36:25.329052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:25.329013    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" event={"ID":"d08d4ddd-ba79-4fe3-899a-619e7b921548","Type":"ContainerStarted","Data":"0571b222e4f930fbe35f8b10c43bfedd90858b944a895121c01bcb6cf6820c14"}
Oct 30 17:36:26.071778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.071748    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"]
Oct 30 17:36:26.083887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.083858    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9824352d-tk6pc"]
Oct 30 17:36:26.270602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.270564    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:26.301533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.301502    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"]
Oct 30 17:36:26.306438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.306415    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.313828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.313797    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df8b20-b98d88686-sqntq\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.313968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.313873    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5wt\" (UniqueName: \"kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt\") pod \"payload-processing-e2e-api-a-df8b20-b98d88686-sqntq\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.341986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.341810    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-df8b20\""
Oct 30 17:36:26.341986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.341957    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-df8b20-dockercfg-tsmrd\""
Oct 30 17:36:26.354892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.348874    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"]
Oct 30 17:36:26.494164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.418216    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df8b20-b98d88686-sqntq\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.494164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.418290    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5wt\" (UniqueName: \"kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt\") pod \"payload-processing-e2e-api-a-df8b20-b98d88686-sqntq\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.494164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.422231    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df8b20-b98d88686-sqntq\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.539724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.539678    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e189b59e-f5a3-4f79-bfe3-2ffa30c264f9" path="/var/lib/kubelet/pods/e189b59e-f5a3-4f79-bfe3-2ffa30c264f9/volumes"
Oct 30 17:36:26.540469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.540442    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" event={"ID":"d08d4ddd-ba79-4fe3-899a-619e7b921548","Type":"ContainerStarted","Data":"50c9df0ccfce8735c1dd6b0371b8c074d2b9ae7a8f5546d23e26c370625c975b"}
Oct 30 17:36:26.541910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.541891    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:26.548469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.547418    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5wt\" (UniqueName: \"kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt\") pod \"payload-processing-e2e-api-a-df8b20-b98d88686-sqntq\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.581145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.579707    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" podStartSLOduration=4.20855297 podStartE2EDuration="4.579691135s" totalImagesPullingTime="371.138165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:26.577920755 +0000 UTC m=+1392.339964568" watchObservedRunningTime="2026-10-30 17:36:26.579691135 +0000 UTC m=+1392.341734948"
Oct 30 17:36:26.623181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.623090    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:26.652549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:26.652516    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:27.025485 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:36:27.025452    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb355b21_bfc6_4ac2_aeb2_a60ce3b448f7.slice/crio-9911fcbaa064ccdc5dcd104ed4e5b7b4a2dbda1d4a723c6ff35840376b165d86 WatchSource:0}: Error finding container 9911fcbaa064ccdc5dcd104ed4e5b7b4a2dbda1d4a723c6ff35840376b165d86: Status 404 returned error can't find the container with id 9911fcbaa064ccdc5dcd104ed4e5b7b4a2dbda1d4a723c6ff35840376b165d86
Oct 30 17:36:27.031095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:27.031068    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"]
Oct 30 17:36:27.366314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:27.366287    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 17:36:27.467944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:27.467906    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq" event={"ID":"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7","Type":"ContainerStarted","Data":"a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e"}
Oct 30 17:36:27.468106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:27.468000    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq" event={"ID":"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7","Type":"ContainerStarted","Data":"9911fcbaa064ccdc5dcd104ed4e5b7b4a2dbda1d4a723c6ff35840376b165d86"}
Oct 30 17:36:27.468195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:27.468181    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:27.518962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:27.518911    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq" podStartSLOduration=1.518897572 podStartE2EDuration="1.518897572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:27.517558955 +0000 UTC m=+1393.279602768" watchObservedRunningTime="2026-10-30 17:36:27.518897572 +0000 UTC m=+1393.280941419"
Oct 30 17:36:28.417893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:28.417862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:28.841757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:28.841724    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:29.445544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:29.445508    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:30.122654 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:30.122597    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:31.122094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:31.122054    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:32.060629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:32.060583    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"]
Oct 30 17:36:32.070027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:32.070002    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44d8871-69dr4"]
Oct 30 17:36:32.376012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:32.375979    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea60540-ed8a-4686-bedf-fb7408634166" path="/var/lib/kubelet/pods/bea60540-ed8a-4686-bedf-fb7408634166/volumes"
Oct 30 17:36:33.459018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:33.458973    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:33.490889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:33.490858    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:36:35.798366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:35.798333    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:36.036825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:36.036790    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:36.565299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:36.565258    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:37.824260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:37.824212    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:39.497202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:39.497161    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:36:40.081306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:40.081269    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:40.978680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:40.978624    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:41.514496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:41.514465    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:42.010381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.010342    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:42.096166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.096134    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"]
Oct 30 17:36:42.104130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.104096    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"]
Oct 30 17:36:42.109602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.109575    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526c403-gpwmq"]
Oct 30 17:36:42.113835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.113808    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658dee5-l2gl9"]
Oct 30 17:36:42.232206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.232164    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:42.410407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.410229    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f8600d-e813-41ab-8829-f01b51a3a521" path="/var/lib/kubelet/pods/c6f8600d-e813-41ab-8829-f01b51a3a521/volumes"
Oct 30 17:36:42.412495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.412474    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec388735-c715-441b-8540-15f79b44f44e" path="/var/lib/kubelet/pods/ec388735-c715-441b-8540-15f79b44f44e/volumes"
Oct 30 17:36:42.474451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:42.474422    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:43.156257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:43.156214    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:44.010624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:44.010569    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:44.147048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:44.147017    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:45.272818 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:36:45.272780    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:36:46.219528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:46.219456    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:47.721630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:47.721578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:47.888860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:47.888825    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:48.495177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:48.495143    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:49.615980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.615944    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:49.794830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.794794    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"]
Oct 30 17:36:49.802443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.802417    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:49.820110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.820090    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-d53ec1-dockercfg-r7plj\""
Oct 30 17:36:49.820859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.820843    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-d53ec1\""
Oct 30 17:36:49.846188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.846162    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:49.880409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.880292    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"]
Oct 30 17:36:49.952669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.931998    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.008944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:49.983370    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:50.009522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.009490    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdsts\" (UniqueName: \"kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts\") pod \"payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:50.049745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.049705    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"]
Oct 30 17:36:50.093788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.093751    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"]
Oct 30 17:36:50.112979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.112940    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdsts\" (UniqueName: \"kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts\") pod \"payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:50.113292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.113271    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npg7j\" (UniqueName: \"kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j\") pod \"payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.113473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.113457    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.113747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.113729    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:50.121071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.121033    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:50.161713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.161607    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdsts\" (UniqueName: \"kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts\") pod \"payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:50.215674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.215613    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npg7j\" (UniqueName: \"kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j\") pod \"payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.215877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.215724    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.216914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.216888    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.256309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.256283    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npg7j\" (UniqueName: \"kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j\") pod \"payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.399739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.399705    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:50.419387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:50.419356    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:51.009231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.009137    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"]
Oct 30 17:36:51.014748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.014519    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"]
Oct 30 17:36:51.018775 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:36:51.018746    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f780ed_4c3c_4952_a45e_5ea8043574c9.slice/crio-ce609f63a411a968978dd78898aac5da52b993f2d6768551335375dd22cf9d32 WatchSource:0}: Error finding container ce609f63a411a968978dd78898aac5da52b993f2d6768551335375dd22cf9d32: Status 404 returned error can't find the container with id ce609f63a411a968978dd78898aac5da52b993f2d6768551335375dd22cf9d32
Oct 30 17:36:51.019930 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:36:51.019905    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990f7d6f_c384_47ec_a6c8_404f64f18d68.slice/crio-cb3ea82fdf70aeb5e7592e018f5c28529df52d73217274a0e3c8ef453b1014f7 WatchSource:0}: Error finding container cb3ea82fdf70aeb5e7592e018f5c28529df52d73217274a0e3c8ef453b1014f7: Status 404 returned error can't find the container with id cb3ea82fdf70aeb5e7592e018f5c28529df52d73217274a0e3c8ef453b1014f7
Oct 30 17:36:51.228244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.228210    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:51.728056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.728017    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" event={"ID":"60f780ed-4c3c-4952-a45e-5ea8043574c9","Type":"ContainerStarted","Data":"052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55"}
Oct 30 17:36:51.728270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.728090    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:36:51.728270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.728114    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" event={"ID":"60f780ed-4c3c-4952-a45e-5ea8043574c9","Type":"ContainerStarted","Data":"ce609f63a411a968978dd78898aac5da52b993f2d6768551335375dd22cf9d32"}
Oct 30 17:36:51.730303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.730266    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" event={"ID":"990f7d6f-c384-47ec-a6c8-404f64f18d68","Type":"ContainerStarted","Data":"5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4"}
Oct 30 17:36:51.730437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.730325    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" event={"ID":"990f7d6f-c384-47ec-a6c8-404f64f18d68","Type":"ContainerStarted","Data":"cb3ea82fdf70aeb5e7592e018f5c28529df52d73217274a0e3c8ef453b1014f7"}
Oct 30 17:36:51.731174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.731148    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:36:51.947667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:51.947367    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" podStartSLOduration=2.947344431 podStartE2EDuration="2.947344431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:51.791288405 +0000 UTC m=+1417.553332219" watchObservedRunningTime="2026-10-30 17:36:51.947344431 +0000 UTC m=+1417.709388244"
Oct 30 17:36:53.043669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:53.040095    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" podStartSLOduration=4.040070995 podStartE2EDuration="4.040070995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:51.983178575 +0000 UTC m=+1417.745222415" watchObservedRunningTime="2026-10-30 17:36:53.040070995 +0000 UTC m=+1418.802114811"
Oct 30 17:36:53.062704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:53.058704    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"]
Oct 30 17:36:53.129056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:53.128993    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5736bb01-9qpg7"]
Oct 30 17:36:53.588665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:53.580222    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:54.137828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.137777    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:54.205668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.204773    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:54.490726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.490325    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:54.519669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.518657    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a" path="/var/lib/kubelet/pods/0eb2dc2f-26db-4b86-b69d-1f7b6ca12b1a/volumes"
Oct 30 17:36:54.653865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.653824    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:36:54.900669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.899887    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 17:36:54.985874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.985830    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:36:54.986475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:54.986443    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:36:55.072666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.071987    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"]
Oct 30 17:36:55.113668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.110479    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d8c5cc-sdd9r"]
Oct 30 17:36:55.138769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.138725    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:55.141033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.139702    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:36:55.180849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.180749    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:55.493204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.493109    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:36:55.584498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.504755    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:55.585290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.583458    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 17:36:55.585763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.585742    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:36:55.655097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.655055    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:55.680669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.655928    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:36:55.784381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.784271    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:55.910860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.910810    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.008668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:55.912956    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:36:56.023727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.010416    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:36:56.199669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.020835    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.320554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.173736    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.321243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.173869    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:36:56.407842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.199615    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.408774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.332043    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:36:56.512341 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.512241    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.594852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.594798    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.596517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.596484    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:56.597777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.597753    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:36:56.657067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.657029    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:36:56.696580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.696541    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd0e7ac-603f-4ef3-8209-93563d372317" path="/var/lib/kubelet/pods/8dd0e7ac-603f-4ef3-8209-93563d372317/volumes"
Oct 30 17:36:56.721741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.697317    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 17:36:56.722037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722016    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:36:56.722192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722180    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:36:56.722328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722317    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 17:36:56.722523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722509    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 17:36:56.722692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722679    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 17:36:56.722833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722822    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:36:56.722953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:56.722943    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:36:57.261680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.255105    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 17:36:57.395145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.395113    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 17:36:57.419609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.419571    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:36:57.424958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.424930    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:36:57.432375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.432349    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:57.527527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.527442    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"]
Oct 30 17:36:57.536669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.536209    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:57.577979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.577044    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 17:36:57.577979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.577096    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx"
Oct 30 17:36:57.577979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.577188    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.621870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.621817    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:36:57.625257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.625228    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"]
Oct 30 17:36:57.631713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.626562    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-df8b20-serving-cert\""
Oct 30 17:36:57.633526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.633502    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:57.636902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.636848    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 17:36:57.643706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.643057    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 17:36:57.743087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.743043    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x924m\" (UniqueName: \"kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.743332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.743160    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.743440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.743386    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.743502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.743490    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.847484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.846754    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x924m\" (UniqueName: \"kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.847779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.847756    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.848408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.848385    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.848751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.848726    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.849308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.849280    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.852020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.851982    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.922666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.919802    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:57.923543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:57.923485    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x924m\" (UniqueName: \"kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m\") pod \"maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:58.018328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:58.018293    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:58.932698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:58.923532    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"]
Oct 30 17:36:59.350265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:59.350231    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:36:59.797150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:59.797107    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s" event={"ID":"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5","Type":"ContainerStarted","Data":"5961bd03e088c1cb9ab23dc455773d28a316462efb1804a8cc54b99eb53dc70c"}
Oct 30 17:36:59.797390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:59.797227    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s" event={"ID":"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5","Type":"ContainerStarted","Data":"382e1a896ef8689884cba0a7b6ef2ec64229ddc163306992cc96738940c62a1c"}
Oct 30 17:36:59.797390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:59.797265    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:36:59.843613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:36:59.843553    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s" podStartSLOduration=3.302426163 podStartE2EDuration="3.843536786s" totalImagesPullingTime="541.110623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:59.842925763 +0000 UTC m=+1425.604969601" watchObservedRunningTime="2026-10-30 17:36:59.843536786 +0000 UTC m=+1425.605580600"
Oct 30 17:37:01.313556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:01.313506    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:02.775873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:02.775841    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 17:37:03.626026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:03.625988    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:03.860557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:03.860518    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:03.970559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:03.970474    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 17:37:05.669411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:05.669374    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:05.813186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:05.813158    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:37:06.131798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.131762    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:37:06.131994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.131832    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:37:06.874422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.874385    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerID="b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f" exitCode=0
Oct 30 17:37:06.874971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.874459    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" event={"ID":"2f7cc5c1-f70f-4fd6-a347-5b40db00e160","Type":"ContainerDied","Data":"b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"}
Oct 30 17:37:06.875066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.875049    5534 scope.go:122] "RemoveContainer" containerID="b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"
Oct 30 17:37:06.876706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.876683    5534 generic.go:356] "Generic (PLEG): container finished" podID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerID="db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8" exitCode=0
Oct 30 17:37:06.876804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.876755    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" event={"ID":"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1","Type":"ContainerDied","Data":"db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"}
Oct 30 17:37:06.877132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.877101    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:06.877173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.877134    5534 scope.go:122] "RemoveContainer" containerID="db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"
Oct 30 17:37:06.877208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.877190    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:06.877256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.877241    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:06.878950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.878920    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:06.879024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.879002    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:06.879061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:06.879045    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:07.739087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:07.739036    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:07.899169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:07.899127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" event={"ID":"2f7cc5c1-f70f-4fd6-a347-5b40db00e160","Type":"ContainerStarted","Data":"14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470"}
Oct 30 17:37:07.902625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:07.902604    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" event={"ID":"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1","Type":"ContainerStarted","Data":"93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e"}
Oct 30 17:37:08.112326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:08.112274    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:37:08.112551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:08.112413    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:37:08.112551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:08.112463    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:37:08.314731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:08.314699    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:37:08.315241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:08.315216    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:37:08.315415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:08.315388    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:37:09.113213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:09.113158    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:37:09.113770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:09.113228    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:37:09.308670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:09.307109    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:37:09.308670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:09.307172    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:37:09.925369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:09.925324    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:37:09.950027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:09.949997    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:10.124668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:10.124100    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:37:10.124668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:10.124165    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:37:10.318259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:10.318167    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:37:10.318523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:10.318494    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:37:11.115140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:11.115100    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:37:11.115434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:11.115405    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:37:11.307201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:11.307162    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:37:11.307876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:11.307835    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:37:12.116668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:12.115156    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:37:12.116668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:12.115222    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:37:12.312668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:12.312143    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:37:12.312668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:12.312213    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:37:12.511661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:12.511549    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:13.068924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.068860    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"]
Oct 30 17:37:13.113670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.113171    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:37:13.113670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.113234    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:37:13.129224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.126690    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"]
Oct 30 17:37:13.129224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.129146    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f3239b-q7sh8"]
Oct 30 17:37:13.134255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.134227    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15de0f8e-72vlt"]
Oct 30 17:37:13.308034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.307998    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:37:13.308306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:13.308279    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:37:14.343735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.342995    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:37:14.390473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.390437    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c4ad86-350d-41a1-aa26-561fa6f6167e" path="/var/lib/kubelet/pods/b2c4ad86-350d-41a1-aa26-561fa6f6167e/volumes"
Oct 30 17:37:14.391461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.391432    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1208ff7-e011-4ceb-83f1-0e269b9c5e9c" path="/var/lib/kubelet/pods/c1208ff7-e011-4ceb-83f1-0e269b9c5e9c/volumes"
Oct 30 17:37:14.392184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.392163    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:37:14.437192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.436141    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:37:14.437721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.437690    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:37:14.466879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.466389    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 17:37:14.959630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:14.959581    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:15.050454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:15.050421    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"]
Oct 30 17:37:15.057087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:15.057054    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6644fcd4-b6zhh"]
Oct 30 17:37:15.167279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:15.167251    5534 scope.go:122] "RemoveContainer" containerID="961234c102661e2c331d1ba4ac8648328b6d2c7d874d53f26860af43b3a72b67"
Oct 30 17:37:15.180185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:15.180165    5534 scope.go:122] "RemoveContainer" containerID="dde9eb800cbfa28edf9d6ff28784058232657536d011b135eaa22b2a4e0e010c"
Oct 30 17:37:15.194482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:15.194452    5534 scope.go:122] "RemoveContainer" containerID="cf3d237e0e19932293b9b12ae701df66cfd6c903b65a3ce2e2fec69dc5495b29"
Oct 30 17:37:15.208595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:15.208570    5534 scope.go:122] "RemoveContainer" containerID="35f2a4b241be9736887bd2ae368ad2baf09dd3bc8bcdbd820e0e809fb21c2ee6"
Oct 30 17:37:15.258524 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:37:15.258487    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:37:16.384540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:16.384504    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f489f6a-7671-4e5a-9b84-f8d9344fd718" path="/var/lib/kubelet/pods/3f489f6a-7671-4e5a-9b84-f8d9344fd718/volumes"
Oct 30 17:37:16.774134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:16.774049    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:18.585865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:18.585825    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:20.403814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:20.403760    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:22.462852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:22.462791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:22.594808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:22.594776    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:22.921281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:22.921235    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:23.037288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:23.037258    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"]
Oct 30 17:37:23.048368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:23.048340    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f3d67-dzqnk"]
Oct 30 17:37:23.057171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:23.057100    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:24.051882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:24.051847    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"]
Oct 30 17:37:24.067316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:24.067282    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caa8ce0-scc6g"]
Oct 30 17:37:24.377831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:24.377796    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135bc2e4-6bd5-46a2-b7bc-c13c429e5731" path="/var/lib/kubelet/pods/135bc2e4-6bd5-46a2-b7bc-c13c429e5731/volumes"
Oct 30 17:37:24.378396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:24.378374    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532b81fe-161a-4f80-bd45-ffd994fb2941" path="/var/lib/kubelet/pods/532b81fe-161a-4f80-bd45-ffd994fb2941/volumes"
Oct 30 17:37:24.526848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:24.526762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:24.776658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:24.776562    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:25.186713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:25.186613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:26.654254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:26.654218    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:28.038227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.038196    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"]
Oct 30 17:37:28.050104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.050068    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"]
Oct 30 17:37:28.054675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.054614    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f233b398-qp95f"]
Oct 30 17:37:28.058422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.058400    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab573e5-fsn5q"]
Oct 30 17:37:28.387960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.387922    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dcc52e-6e49-4eb5-baef-f3f589159bc7" path="/var/lib/kubelet/pods/67dcc52e-6e49-4eb5-baef-f3f589159bc7/volumes"
Oct 30 17:37:28.388587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.388570    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c8a1ef-b710-4c57-b1c9-7a721202f156" path="/var/lib/kubelet/pods/c0c8a1ef-b710-4c57-b1c9-7a721202f156/volumes"
Oct 30 17:37:28.545491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:28.545458    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:30.331735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:30.331696    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:30.478387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:30.478352    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:32.523458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:32.523424    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:33.950341 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:33.950300    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:34.701845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:34.701812    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:34.885939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:34.885908    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:35.043487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:35.043398    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"]
Oct 30 17:37:35.054885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:35.054854    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f217ac-wtwr7"]
Oct 30 17:37:35.246381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:35.246345    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:35.713935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:35.713903    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:35.885668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:35.884634    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:36.374445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:36.374410    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdf13e4-092a-456e-9f44-180f3066103b" path="/var/lib/kubelet/pods/5fdf13e4-092a-456e-9f44-180f3066103b/volumes"
Oct 30 17:37:36.439834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:36.439797    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:36.550930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:36.550890    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:36.998029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:36.998001    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:37.137868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:37.137837    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:38.342073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.342040    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:38.640318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.640246    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"]
Oct 30 17:37:38.645051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.645033    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.662037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.662016    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81e9336e-openshift-default-dockercfg-4mbp4\""
Oct 30 17:37:38.681971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.681938    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"]
Oct 30 17:37:38.720294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720239    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720363    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720484    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720509    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720546    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720756    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2845\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720884    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.720985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720928    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.721044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.720997    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822323    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822445    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822480    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822515    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822592    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2845\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822728    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822768    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.822841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.822838    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.823132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.823039    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.823699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.823258    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.823699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.823400    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.823699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.823584    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.823699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.823634    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.824026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.824003    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.825152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.825124    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.825356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.825339    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.846582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.846544    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2845\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.846762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.846700    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-cw26f\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:38.964557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:38.964454    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:39.305866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:39.305835    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"]
Oct 30 17:37:39.308949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:39.308912    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:39.309064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:39.308983    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:39.309064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:39.309014    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:37:40.138134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.138100    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:40.328199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.328163    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" event={"ID":"2d80f646-adfc-4423-9d74-af2f9ef75990","Type":"ContainerStarted","Data":"f6d924aa61376269ba9d007f717ca66ddba7b1d2289a35a3a1dc2d7bb2dab684"}
Oct 30 17:37:40.328370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.328222    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" event={"ID":"2d80f646-adfc-4423-9d74-af2f9ef75990","Type":"ContainerStarted","Data":"57090ab6b924c7b907a3fd4a834f3ad71dabe7e94beb4ea787fc0bb6a4c99cdb"}
Oct 30 17:37:40.401056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.400920    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podStartSLOduration=2.400899952 podStartE2EDuration="2.400899952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:40.39954067 +0000 UTC m=+1466.161584484" watchObservedRunningTime="2026-10-30 17:37:40.400899952 +0000 UTC m=+1466.162943766"
Oct 30 17:37:40.973670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.968838    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:40.973670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.970386    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 17:37:40.973670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:40.970443    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 17:37:41.094838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:41.094773    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"]
Oct 30 17:37:41.100696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:41.099879    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d250889-lzs47"]
Oct 30 17:37:41.975669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:41.971974    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 17:37:41.975669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:41.972030    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 17:37:41.985666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:41.985451    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:42.400505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:42.400466    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5859cd-3152-4a76-bd22-47ae1f877556" path="/var/lib/kubelet/pods/cd5859cd-3152-4a76-bd22-47ae1f877556/volumes"
Oct 30 17:37:42.969031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:42.968995    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 17:37:42.969324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:42.969291    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 17:37:43.085633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:43.085591    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"]
Oct 30 17:37:43.091730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:43.091705    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490b2856-5k972"]
Oct 30 17:37:43.965014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:43.964856    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 17:37:43.965014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:43.964918    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 17:37:44.103298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:44.103262    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:44.398849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:44.398812    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc4ea5a-d1e2-4007-91cc-53fb81272f97" path="/var/lib/kubelet/pods/2fc4ea5a-d1e2-4007-91cc-53fb81272f97/volumes"
Oct 30 17:37:44.923090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:44.923036    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:44.974771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:44.974732    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 17:37:44.975099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:44.975069    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 17:37:45.248010 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:37:45.247920    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:37:45.957034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:45.956992    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:45.970182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:45.970143    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 17:37:45.970377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:45.970212    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 17:37:46.969776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:46.969746    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:46.970291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:46.970107    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:46.971065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:46.971044    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:37:47.710923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:47.710884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:49.699581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:49.699533    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:50.335297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:50.335256    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:50.596862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:50.596792    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:51.018012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:51.017876    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:51.197696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:51.196144    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:52.181623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:52.181581    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:52.350894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:52.350858    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:52.530935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:52.530821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:54.050266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:54.050233    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"]
Oct 30 17:37:54.060239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:54.060203    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2600ca07-8hqs7"]
Oct 30 17:37:54.205424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:54.205387    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:54.372838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:54.372806    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fa06d9-3f50-45fe-92c1-2d437c0fb543" path="/var/lib/kubelet/pods/80fa06d9-3f50-45fe-92c1-2d437c0fb543/volumes"
Oct 30 17:37:54.373054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:54.372901    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:54.668583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:54.668494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:55.037981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:55.037881    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"]
Oct 30 17:37:55.050263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:55.050226    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbc5c0-jmdfl"]
Oct 30 17:37:55.545146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:55.545114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:56.394665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:56.394002    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb630fd3-c4e7-463a-966e-8d99679bcef3" path="/var/lib/kubelet/pods/bb630fd3-c4e7-463a-966e-8d99679bcef3/volumes"
Oct 30 17:37:56.405667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:56.400783    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:58.235707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.235630    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:37:58.831946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.831908    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"]
Oct 30 17:37:58.839397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.839372    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.859160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.859124    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"]
Oct 30 17:37:58.863385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.863355    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:37:58.863755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.863730    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:37:58.867802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.867754    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.868010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.867985    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.868458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.868435    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczqs\" (UniqueName: \"kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.970849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.970814    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczqs\" (UniqueName: \"kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.971062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.970921    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.971134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.971061    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.971704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.971678    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:58.973808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:58.973780    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:59.001277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:59.001244    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczqs\" (UniqueName: \"kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs\") pod \"maas-api-revoke-sub-8dcef238-m2sfh\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:59.156933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:59.156901    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:37:59.495149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:59.495040    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"]
Oct 30 17:37:59.646157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:37:59.646126    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh" event={"ID":"740c0b49-5700-4b79-9f71-508227c47790","Type":"ContainerStarted","Data":"0150d2728e655a734ee65c20279416ad6d41a9de6023b51ca0d90071e3fe0599"}
Oct 30 17:38:00.046421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:00.046386    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"]
Oct 30 17:38:00.060286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:00.060249    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e84683-vlhqf"]
Oct 30 17:38:00.217033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:00.216945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:00.378283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:00.378250    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3499c804-eb7e-4036-bec8-bbf10fd28082" path="/var/lib/kubelet/pods/3499c804-eb7e-4036-bec8-bbf10fd28082/volumes"
Oct 30 17:38:00.671692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:00.657752    5534 generic.go:356] "Generic (PLEG): container finished" podID="740c0b49-5700-4b79-9f71-508227c47790" containerID="e35908f523d73082728832c1055746ffaaff7e3752bec86835723d6425b2742e" exitCode=0
Oct 30 17:38:00.671692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:00.658058    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh" event={"ID":"740c0b49-5700-4b79-9f71-508227c47790","Type":"ContainerDied","Data":"e35908f523d73082728832c1055746ffaaff7e3752bec86835723d6425b2742e"}
Oct 30 17:38:01.865637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.865603    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:38:01.927373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.927273    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca\") pod \"740c0b49-5700-4b79-9f71-508227c47790\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") "
Oct 30 17:38:01.927728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.927704    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca\") pod \"740c0b49-5700-4b79-9f71-508227c47790\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") "
Oct 30 17:38:01.927880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.927840    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs\" (UniqueName: \"kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs\") pod \"740c0b49-5700-4b79-9f71-508227c47790\" (UID: \"740c0b49-5700-4b79-9f71-508227c47790\") "
Oct 30 17:38:01.933289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.933167    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs" pod "740c0b49-5700-4b79-9f71-508227c47790" (UID: "740c0b49-5700-4b79-9f71-508227c47790"). InnerVolumeSpecName "kube-api-access-kczqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:01.935266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.933906    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca" pod "740c0b49-5700-4b79-9f71-508227c47790" (UID: "740c0b49-5700-4b79-9f71-508227c47790"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:01.950925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.950868    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca" pod "740c0b49-5700-4b79-9f71-508227c47790" (UID: "740c0b49-5700-4b79-9f71-508227c47790"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:38:01.986053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:01.986013    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:02.031586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:02.031551    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740c0b49-5700-4b79-9f71-508227c47790-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:02.031586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:02.031585    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740c0b49-5700-4b79-9f71-508227c47790-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:02.031833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:02.031600    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kczqs\" (UniqueName: \"kubernetes.io/projected/740c0b49-5700-4b79-9f71-508227c47790-kube-api-access-kczqs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:02.682575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:02.682479    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"
Oct 30 17:38:02.682575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:02.682487    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh" event={"ID":"740c0b49-5700-4b79-9f71-508227c47790","Type":"ContainerDied","Data":"0150d2728e655a734ee65c20279416ad6d41a9de6023b51ca0d90071e3fe0599"}
Oct 30 17:38:02.682575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:02.682551    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0150d2728e655a734ee65c20279416ad6d41a9de6023b51ca0d90071e3fe0599"
Oct 30 17:38:03.649844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:03.649802    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:05.090579 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.090545    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"]
Oct 30 17:38:05.091537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.091514    5534 cpu_manager.go:414] "containerMap: removing container" podUID="740c0b49-5700-4b79-9f71-508227c47790" containerName="revoke-keys"
Oct 30 17:38:05.091691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.091538    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740c0b49-5700-4b79-9f71-508227c47790" containerName="revoke-keys"
Oct 30 17:38:05.091810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.091795    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="740c0b49-5700-4b79-9f71-508227c47790" containerName="revoke-keys"
Oct 30 17:38:05.095025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.094993    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.123278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.123244    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:38:05.123439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.123295    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:38:05.132009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.131979    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"]
Oct 30 17:38:05.195689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.195632    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.199941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.199385    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.199941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.199444    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4d2\" (UniqueName: \"kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.301828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.301786    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.302043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.301885    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.302043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.301919    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4d2\" (UniqueName: \"kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.302748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.302719    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.304821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.304792    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.325393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.325370    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4d2\" (UniqueName: \"kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-nff5s\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.421457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.421364    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:05.738608 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:38:05.738569    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61071055_3b08_4cfc_967a_be7b35d88aa1.slice/crio-d645695b84c672bea12cb513b4998f4ca05184a328fa95a07ed82ff2a820cbab WatchSource:0}: Error finding container d645695b84c672bea12cb513b4998f4ca05184a328fa95a07ed82ff2a820cbab: Status 404 returned error can't find the container with id d645695b84c672bea12cb513b4998f4ca05184a328fa95a07ed82ff2a820cbab
Oct 30 17:38:05.738824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.738613    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"]
Oct 30 17:38:05.885347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:05.885310    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:06.728970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:06.728928    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s" event={"ID":"61071055-3b08-4cfc-967a-be7b35d88aa1","Type":"ContainerStarted","Data":"58dcae5863c32c6d655526ddee9d301828508d237a27f5d31c88b11e4d0e0323"}
Oct 30 17:38:06.729466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:06.729002    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s" event={"ID":"61071055-3b08-4cfc-967a-be7b35d88aa1","Type":"ContainerStarted","Data":"d645695b84c672bea12cb513b4998f4ca05184a328fa95a07ed82ff2a820cbab"}
Oct 30 17:38:06.773830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:06.773758    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s" podStartSLOduration=1.773738093 podStartE2EDuration="1.773738093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:06.769012058 +0000 UTC m=+1492.531055871" watchObservedRunningTime="2026-10-30 17:38:06.773738093 +0000 UTC m=+1492.535781924"
Oct 30 17:38:07.728065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:07.727997    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:07.741022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:07.740975    5534 generic.go:356] "Generic (PLEG): container finished" podID="61071055-3b08-4cfc-967a-be7b35d88aa1" containerID="58dcae5863c32c6d655526ddee9d301828508d237a27f5d31c88b11e4d0e0323" exitCode=0
Oct 30 17:38:07.741545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:07.741127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s" event={"ID":"61071055-3b08-4cfc-967a-be7b35d88aa1","Type":"ContainerDied","Data":"58dcae5863c32c6d655526ddee9d301828508d237a27f5d31c88b11e4d0e0323"}
Oct 30 17:38:08.935028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:08.935004    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:09.077349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.077268    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca\") pod \"61071055-3b08-4cfc-967a-be7b35d88aa1\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") "
Oct 30 17:38:09.077534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.077379    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca\") pod \"61071055-3b08-4cfc-967a-be7b35d88aa1\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") "
Oct 30 17:38:09.077681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.077633    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2\" (UniqueName: \"kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2\") pod \"61071055-3b08-4cfc-967a-be7b35d88aa1\" (UID: \"61071055-3b08-4cfc-967a-be7b35d88aa1\") "
Oct 30 17:38:09.077933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.077898    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca" pod "61071055-3b08-4cfc-967a-be7b35d88aa1" (UID: "61071055-3b08-4cfc-967a-be7b35d88aa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:09.078475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.078447    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:09.084691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.079978    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca" pod "61071055-3b08-4cfc-967a-be7b35d88aa1" (UID: "61071055-3b08-4cfc-967a-be7b35d88aa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:38:09.084691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.080094    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2" pod "61071055-3b08-4cfc-967a-be7b35d88aa1" (UID: "61071055-3b08-4cfc-967a-be7b35d88aa1"). InnerVolumeSpecName "kube-api-access-sg4d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:09.166584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.166553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:09.179931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.179904    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61071055-3b08-4cfc-967a-be7b35d88aa1-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:09.179931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.179932    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4d2\" (UniqueName: \"kubernetes.io/projected/61071055-3b08-4cfc-967a-be7b35d88aa1-kube-api-access-sg4d2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:09.495152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.495081    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:09.763754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.763666    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"
Oct 30 17:38:09.763754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.763679    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s" event={"ID":"61071055-3b08-4cfc-967a-be7b35d88aa1","Type":"ContainerDied","Data":"d645695b84c672bea12cb513b4998f4ca05184a328fa95a07ed82ff2a820cbab"}
Oct 30 17:38:09.763967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:09.763776    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d645695b84c672bea12cb513b4998f4ca05184a328fa95a07ed82ff2a820cbab"
Oct 30 17:38:10.031099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:10.030928    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"]
Oct 30 17:38:10.035601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:10.035574    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096dbee-v8942"]
Oct 30 17:38:10.371995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:10.371954    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aabf6f5-b57e-4168-a888-193f241790b3" path="/var/lib/kubelet/pods/8aabf6f5-b57e-4168-a888-193f241790b3/volumes"
Oct 30 17:38:11.528152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:11.528121    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:13.293180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:13.293145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:15.036997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.036956    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"]
Oct 30 17:38:15.055808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.055780    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8453eb74-svgmp"]
Oct 30 17:38:15.233997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.233968    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:15.245289 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:38:15.245258    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:38:15.278891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.278846    5534 scope.go:122] "RemoveContainer" containerID="6a45e280a00475f6e0c95bd956c4dbb7c4ca23b263f5f099cf3d812ec23c0d87"
Oct 30 17:38:15.313132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.307206    5534 scope.go:122] "RemoveContainer" containerID="e80c2ac1b6b188b35f324f054fb6332e9e9c7846d5e61c0c33217b7d9744e694"
Oct 30 17:38:15.331866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.331830    5534 scope.go:122] "RemoveContainer" containerID="a7e3969ac28da47fecdf4d41e0933c1289f4f768a2467f224dca1b0388a04bfb"
Oct 30 17:38:15.345551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.345532    5534 scope.go:122] "RemoveContainer" containerID="67212b1427563c5d1746cda395d2317ce7b39b9af84921e0c7c7618c25e475dd"
Oct 30 17:38:15.355969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.355947    5534 scope.go:122] "RemoveContainer" containerID="ffe9b7c9f18e0e22cb85b451f4dd63ada00906ab8e87defa722ee8e834c2fcfc"
Oct 30 17:38:15.376131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.376105    5534 scope.go:122] "RemoveContainer" containerID="08227bf1c18a2422bfb9a7dd3090a7640551b2ef4cdf4ff5b49a515c5fefd014"
Oct 30 17:38:15.400763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.398436    5534 scope.go:122] "RemoveContainer" containerID="599ce212844532f60f7690e33244f7f1922c6b606f885a974ed3b8d1c50ed02c"
Oct 30 17:38:15.412959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.412933    5534 scope.go:122] "RemoveContainer" containerID="4bdc6ebc4c9d52d3b38bbc2136449f5a1744543ced87ee33160c70b1bf5aab65"
Oct 30 17:38:15.426014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.425995    5534 scope.go:122] "RemoveContainer" containerID="42e952c79aeee0dcaf2297389059a6b4eb32e9ffeaabd51a637d8a9d334ee48d"
Oct 30 17:38:15.435874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.435854    5534 scope.go:122] "RemoveContainer" containerID="4c94695aa753a1358700c33bc6545c8d6e6c94cb467e7c651251f9a1a755cb7b"
Oct 30 17:38:15.587054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.586978    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"]
Oct 30 17:38:15.742573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.742540    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:15.859913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:15.859886    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:16.283734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:16.283045    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:16.405666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:16.405440    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24accd9e-69f3-47ef-a8d7-2cb1207d6224" path="/var/lib/kubelet/pods/24accd9e-69f3-47ef-a8d7-2cb1207d6224/volumes"
Oct 30 17:38:17.004447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:17.004418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:18.776457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:18.776415    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:19.254314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:19.254277    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:20.397715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:20.395307    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:20.524316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:20.524270    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:20.593623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:20.593494    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerName="payload-pre-processing" containerID="cri-o://7ed2bf3149cd0030eaea5380aed6f5bf036554b09468c6cf79dbcad4ca9112b0" gracePeriod=25
Oct 30 17:38:22.179857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:22.179824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:22.302895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:22.302855    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:22.424836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:22.424794    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:22.784952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:22.784906    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:38:24.115882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:24.115845    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:24.420241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:24.420154    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"]
Oct 30 17:38:24.420496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:24.420410    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy" containerID="cri-o://ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48" gracePeriod=30
Oct 30 17:38:26.175066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:26.174530    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:28.034829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:28.034795    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"]
Oct 30 17:38:28.086480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:28.086450    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1417ef52-d58dl"]
Oct 30 17:38:28.165993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:28.162874    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:28.374943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:28.374908    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cad0190-8798-4785-972f-6801410c9ba9" path="/var/lib/kubelet/pods/5cad0190-8798-4785-972f-6801410c9ba9/volumes"
Oct 30 17:38:29.739195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.739169    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:38:29.779095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779055    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779125    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779250    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779278    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779308    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779396    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779472    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:29.779658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779550    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779613    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.779774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779682    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:29.780068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779974    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:29.780284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.780129    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:29.780675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.780180    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:29.780675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.779713    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7\") pod \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\" (UID: \"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6\") "
Oct 30 17:38:29.781844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.781823    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.782125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782112    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.782264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782250    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.782379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782367    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.782484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782472    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.782576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782077    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:29.782731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782401    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "kube-api-access-x9ll7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:29.782833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.782608    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:29.783167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.783141    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo" pod "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" (UID: "58d8bf3b-e41f-45d4-b3eb-8b1a459178d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:38:29.884382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.884345    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.884382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.884377    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ll7\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-kube-api-access-x9ll7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.884382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.884387    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:29.884759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:29.884399    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:30.021102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.021016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:30.036615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.036582    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"]
Oct 30 17:38:30.047311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.047282    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5714b693-6vsb4"]
Oct 30 17:38:30.082332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.082293    5534 generic.go:356] "Generic (PLEG): container finished" podID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerID="ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48" exitCode=0
Oct 30 17:38:30.082875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.082722    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"
Oct 30 17:38:30.083051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.082747    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" event={"ID":"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6","Type":"ContainerDied","Data":"ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48"}
Oct 30 17:38:30.088589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.088566    5534 scope.go:122] "RemoveContainer" containerID="ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48"
Oct 30 17:38:30.089169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.089142    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82" event={"ID":"58d8bf3b-e41f-45d4-b3eb-8b1a459178d6","Type":"ContainerDied","Data":"213285e74776883d900d8a47a136b7377a86b0987645d0f0106f6a9fa8c178fd"}
Oct 30 17:38:30.114580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.114551    5534 scope.go:122] "RemoveContainer" containerID="ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48"
Oct 30 17:38:30.114987 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:38:30.114961    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48\": container with ID starting with ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48 not found: ID does not exist" containerID="ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48"
Oct 30 17:38:30.115081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.115001    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48"} err="failed to get container status \"ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48\": rpc error: code = NotFound desc = could not find container \"ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48\": container with ID starting with ae89a8dbad8a68910a43e38830e0a421340e5cc24b35b88bbde7706f4a1a1f48 not found: ID does not exist"
Oct 30 17:38:30.148482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.148456    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"]
Oct 30 17:38:30.152439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.152416    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-q9c82"]
Oct 30 17:38:30.372712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.372679    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56963b27-c254-4064-8dd5-519fff13e3c4" path="/var/lib/kubelet/pods/56963b27-c254-4064-8dd5-519fff13e3c4/volumes"
Oct 30 17:38:30.373239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.373224    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" path="/var/lib/kubelet/pods/58d8bf3b-e41f-45d4-b3eb-8b1a459178d6/volumes"
Oct 30 17:38:30.452529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.452499    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:30.766498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.766389    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:30.878934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:30.878896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:31.810568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:31.810534    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:32.784953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:32.784911    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:38:33.478150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:33.478092    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:35.192132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:35.192102    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:36.821104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:36.821073    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:38.286869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.286833    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"]
Oct 30 17:38:38.288059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.288033    5534 cpu_manager.go:414] "containerMap: removing container" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy"
Oct 30 17:38:38.288205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.288061    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy"
Oct 30 17:38:38.288205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.288157    5534 cpu_manager.go:414] "containerMap: removing container" podUID="61071055-3b08-4cfc-967a-be7b35d88aa1" containerName="revoke-keys"
Oct 30 17:38:38.288205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.288166    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61071055-3b08-4cfc-967a-be7b35d88aa1" containerName="revoke-keys"
Oct 30 17:38:38.288376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.288335    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="61071055-3b08-4cfc-967a-be7b35d88aa1" containerName="revoke-keys"
Oct 30 17:38:38.288376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.288351    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d8bf3b-e41f-45d4-b3eb-8b1a459178d6" containerName="istio-proxy"
Oct 30 17:38:38.292865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.292845    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.312084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.312058    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"]
Oct 30 17:38:38.318182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.318155    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:38:38.318375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.318214    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:38:38.340429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.340399    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.340592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.340553    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.340744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.340720    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptbp\" (UniqueName: \"kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.442128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.442082    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.442338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.442216    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptbp\" (UniqueName: \"kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.442465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.442449    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.443103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.443080    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.444725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.444704    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.465923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.465897    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptbp\" (UniqueName: \"kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp\") pod \"maas-api-revoke-sub-ec015ffb-phnk8\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:38.605494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.605459    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:38.608579 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:38.608552    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:39.000277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:39.000249    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"]
Oct 30 17:38:39.000495 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:38:39.000470    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fbb6cf_0168_4619_ac2f_c7335109e2fc.slice/crio-2f2af5efa67174ad493294cf3d264c3a7dd27dbbc29600ed067df0e1985849bc WatchSource:0}: Error finding container 2f2af5efa67174ad493294cf3d264c3a7dd27dbbc29600ed067df0e1985849bc: Status 404 returned error can't find the container with id 2f2af5efa67174ad493294cf3d264c3a7dd27dbbc29600ed067df0e1985849bc
Oct 30 17:38:39.170621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:39.170579    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8" event={"ID":"f5fbb6cf-0168-4619-ac2f-c7335109e2fc","Type":"ContainerStarted","Data":"2f2af5efa67174ad493294cf3d264c3a7dd27dbbc29600ed067df0e1985849bc"}
Oct 30 17:38:39.623273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:39.623242    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"]
Oct 30 17:38:39.623744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:39.623485    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy" containerID="cri-o://a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e" gracePeriod=30
Oct 30 17:38:40.038441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:40.038398    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"]
Oct 30 17:38:40.047315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:40.047272    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c588e8b-pwqlc"]
Oct 30 17:38:40.184069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:40.184039    5534 generic.go:356] "Generic (PLEG): container finished" podID="f5fbb6cf-0168-4619-ac2f-c7335109e2fc" containerID="7a3afdf05cb3adbcca4be479fd00bc115a41597929fa18e33b5c0c896b3160ef" exitCode=0
Oct 30 17:38:40.184273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:40.184076    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8" event={"ID":"f5fbb6cf-0168-4619-ac2f-c7335109e2fc","Type":"ContainerDied","Data":"7a3afdf05cb3adbcca4be479fd00bc115a41597929fa18e33b5c0c896b3160ef"}
Oct 30 17:38:40.373967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:40.373936    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3127ff94-efc3-4699-bfdf-0f38fcccf2fd" path="/var/lib/kubelet/pods/3127ff94-efc3-4699-bfdf-0f38fcccf2fd/volumes"
Oct 30 17:38:40.505438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:40.505405    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:41.195820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.195789    5534 generic.go:356] "Generic (PLEG): container finished" podID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerID="7ed2bf3149cd0030eaea5380aed6f5bf036554b09468c6cf79dbcad4ca9112b0" exitCode=0
Oct 30 17:38:41.196297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.195865    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" event={"ID":"777dd8b3-39cb-4ef2-9980-fccaa02d3650","Type":"ContainerDied","Data":"7ed2bf3149cd0030eaea5380aed6f5bf036554b09468c6cf79dbcad4ca9112b0"}
Oct 30 17:38:41.339792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.339566    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:38:41.443289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.443258    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:41.528272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.528245    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft\" (UniqueName: \"kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft\") pod \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") "
Oct 30 17:38:41.528493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.528303    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca\") pod \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") "
Oct 30 17:38:41.528493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.528460    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca\") pod \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") "
Oct 30 17:38:41.528806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.528618    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume\") pod \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\" (UID: \"777dd8b3-39cb-4ef2-9980-fccaa02d3650\") "
Oct 30 17:38:41.528806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.528738    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp\" (UniqueName: \"kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp\") pod \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\" (UID: \"f5fbb6cf-0168-4619-ac2f-c7335109e2fc\") "
Oct 30 17:38:41.529281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.529040    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume" pod "777dd8b3-39cb-4ef2-9980-fccaa02d3650" (UID: "777dd8b3-39cb-4ef2-9980-fccaa02d3650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:41.529811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.529601    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777dd8b3-39cb-4ef2-9980-fccaa02d3650-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:41.530083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.530010    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca" pod "f5fbb6cf-0168-4619-ac2f-c7335109e2fc" (UID: "f5fbb6cf-0168-4619-ac2f-c7335109e2fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:41.531275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.531252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca" pod "f5fbb6cf-0168-4619-ac2f-c7335109e2fc" (UID: "f5fbb6cf-0168-4619-ac2f-c7335109e2fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:38:41.531368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.531345    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp" pod "f5fbb6cf-0168-4619-ac2f-c7335109e2fc" (UID: "f5fbb6cf-0168-4619-ac2f-c7335109e2fc"). InnerVolumeSpecName "kube-api-access-5ptbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:41.531435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.531416    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft" pod "777dd8b3-39cb-4ef2-9980-fccaa02d3650" (UID: "777dd8b3-39cb-4ef2-9980-fccaa02d3650"). InnerVolumeSpecName "kube-api-access-xcdft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:41.631457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.631428    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdft\" (UniqueName: \"kubernetes.io/projected/777dd8b3-39cb-4ef2-9980-fccaa02d3650-kube-api-access-xcdft\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:41.631457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.631457    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:41.631457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.631466    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:41.631783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.631478    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptbp\" (UniqueName: \"kubernetes.io/projected/f5fbb6cf-0168-4619-ac2f-c7335109e2fc-kube-api-access-5ptbp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:41.956166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:41.956123    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:42.077485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.077447    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:42.201547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.201507    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:42.209659 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.209568    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"
Oct 30 17:38:42.209793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.209563    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8" event={"ID":"f5fbb6cf-0168-4619-ac2f-c7335109e2fc","Type":"ContainerDied","Data":"2f2af5efa67174ad493294cf3d264c3a7dd27dbbc29600ed067df0e1985849bc"}
Oct 30 17:38:42.209793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.209721    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2af5efa67174ad493294cf3d264c3a7dd27dbbc29600ed067df0e1985849bc"
Oct 30 17:38:42.211170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.211149    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"
Oct 30 17:38:42.211170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.211159    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27" event={"ID":"777dd8b3-39cb-4ef2-9980-fccaa02d3650","Type":"ContainerDied","Data":"0a4035bb9fd6a3ac057e7f18fb10ba764c708f43873c7526a8efed2c2a1ae402"}
Oct 30 17:38:42.211323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.211213    5534 scope.go:122] "RemoveContainer" containerID="7ed2bf3149cd0030eaea5380aed6f5bf036554b09468c6cf79dbcad4ca9112b0"
Oct 30 17:38:42.291489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.291458    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"]
Oct 30 17:38:42.298351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.298315    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-mwr27"]
Oct 30 17:38:42.345880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.345850    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:42.372686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:42.372629    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" path="/var/lib/kubelet/pods/777dd8b3-39cb-4ef2-9980-fccaa02d3650/volumes"
Oct 30 17:38:43.831255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.831212    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"]
Oct 30 17:38:43.832919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.832890    5534 cpu_manager.go:414] "containerMap: removing container" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerName="payload-pre-processing"
Oct 30 17:38:43.832919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.832918    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerName="payload-pre-processing"
Oct 30 17:38:43.833104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.832944    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f5fbb6cf-0168-4619-ac2f-c7335109e2fc" containerName="revoke-keys"
Oct 30 17:38:43.833104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.832951    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fbb6cf-0168-4619-ac2f-c7335109e2fc" containerName="revoke-keys"
Oct 30 17:38:43.833302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.833283    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="777dd8b3-39cb-4ef2-9980-fccaa02d3650" containerName="payload-pre-processing"
Oct 30 17:38:43.833357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.833315    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fbb6cf-0168-4619-ac2f-c7335109e2fc" containerName="revoke-keys"
Oct 30 17:38:43.840670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.840629    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.852218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.852183    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"]
Oct 30 17:38:43.859196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.859176    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:38:43.859370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.859207    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:38:43.869359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.869329    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6trk\" (UniqueName: \"kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.869489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.869384    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.869751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.869725    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.971216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.971181    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.971389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.971372    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6trk\" (UniqueName: \"kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.971472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.971409    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.972068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.972048    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.973792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.973770    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:43.984896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.984865    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:43.995782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:43.995749    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6trk\" (UniqueName: \"kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk\") pod \"maas-api-revoke-sub-abd4e563-7qzgg\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:44.159169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.159131    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:44.476612 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:38:44.476583    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635e8b8c_0323_49ee_95c5_204b7baa0920.slice/crio-3edbd3709ffa37131620dba6d07a63ee9e73aee5220ef0290f04a6daf6de8b3f WatchSource:0}: Error finding container 3edbd3709ffa37131620dba6d07a63ee9e73aee5220ef0290f04a6daf6de8b3f: Status 404 returned error can't find the container with id 3edbd3709ffa37131620dba6d07a63ee9e73aee5220ef0290f04a6daf6de8b3f
Oct 30 17:38:44.477016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.476996    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"]
Oct 30 17:38:44.871479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.871444    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:44.901425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.901401    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:38:44.990578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990535    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.990835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990666    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.990835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990708    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.990835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990768    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.990835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990806    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.991089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990905    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.991089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.990953    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.991089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.991008    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.991254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.991116    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy\") pod \"1444c317-574f-48cd-a200-06c343b2d5ce\" (UID: \"1444c317-574f-48cd-a200-06c343b2d5ce\") "
Oct 30 17:38:44.995026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.994959    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:44.995962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.995741    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:44.995962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.995966    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:44.996665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.996189    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:44.997008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.996957    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:44.997098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.997040    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:44.998724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.998696    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:38:44.999119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.999097    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:44.999533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:44.999509    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj" pod "1444c317-574f-48cd-a200-06c343b2d5ce" (UID: "1444c317-574f-48cd-a200-06c343b2d5ce"). InnerVolumeSpecName "kube-api-access-vf2mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:45.016609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.016569    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:45.093729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093691    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093727    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093741    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093754    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1444c317-574f-48cd-a200-06c343b2d5ce-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093768    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1444c317-574f-48cd-a200-06c343b2d5ce-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093780    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093793    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093808    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1444c317-574f-48cd-a200-06c343b2d5ce-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.093945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.093822    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2mj\" (UniqueName: \"kubernetes.io/projected/1444c317-574f-48cd-a200-06c343b2d5ce-kube-api-access-vf2mj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:45.245924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.245836    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg" event={"ID":"635e8b8c-0323-49ee-95c5-204b7baa0920","Type":"ContainerStarted","Data":"7355bf2fb14cc3a0a808e5db49721ae58c0ba6e9fbe62476a379a592166b5f35"}
Oct 30 17:38:45.245924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.245902    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg" event={"ID":"635e8b8c-0323-49ee-95c5-204b7baa0920","Type":"ContainerStarted","Data":"3edbd3709ffa37131620dba6d07a63ee9e73aee5220ef0290f04a6daf6de8b3f"}
Oct 30 17:38:45.247952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.247920    5534 generic.go:356] "Generic (PLEG): container finished" podID="1444c317-574f-48cd-a200-06c343b2d5ce" containerID="a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e" exitCode=0
Oct 30 17:38:45.248097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.247978    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"
Oct 30 17:38:45.248097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.248005    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" event={"ID":"1444c317-574f-48cd-a200-06c343b2d5ce","Type":"ContainerDied","Data":"a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e"}
Oct 30 17:38:45.248097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.248057    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn" event={"ID":"1444c317-574f-48cd-a200-06c343b2d5ce","Type":"ContainerDied","Data":"7dd338e10b8376dff92f345d0ae065ad2dbca1c9c1b8c0aac46dea6ae6291d70"}
Oct 30 17:38:45.248097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.248093    5534 scope.go:122] "RemoveContainer" containerID="a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e"
Oct 30 17:38:45.251814 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:38:45.251787    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:38:45.260288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.260270    5534 scope.go:122] "RemoveContainer" containerID="a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e"
Oct 30 17:38:45.260584 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:38:45.260562    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e\": container with ID starting with a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e not found: ID does not exist" containerID="a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e"
Oct 30 17:38:45.260661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.260597    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e"} err="failed to get container status \"a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e\": rpc error: code = NotFound desc = could not find container \"a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e\": container with ID starting with a9c1372d8169aad9abc3b49e96b13c0bd0f318ff7544cefeb41fee291a9dd05e not found: ID does not exist"
Oct 30 17:38:45.292921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.292862    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg" podStartSLOduration=2.292846407 podStartE2EDuration="2.292846407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:45.28669874 +0000 UTC m=+1531.048742573" watchObservedRunningTime="2026-10-30 17:38:45.292846407 +0000 UTC m=+1531.054890222"
Oct 30 17:38:45.328027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.327991    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"]
Oct 30 17:38:45.339094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.339068    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-jbwhn"]
Oct 30 17:38:45.807241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:45.807088    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:46.258607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:46.258573    5534 generic.go:356] "Generic (PLEG): container finished" podID="635e8b8c-0323-49ee-95c5-204b7baa0920" containerID="7355bf2fb14cc3a0a808e5db49721ae58c0ba6e9fbe62476a379a592166b5f35" exitCode=0
Oct 30 17:38:46.259105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:46.258679    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg" event={"ID":"635e8b8c-0323-49ee-95c5-204b7baa0920","Type":"ContainerDied","Data":"7355bf2fb14cc3a0a808e5db49721ae58c0ba6e9fbe62476a379a592166b5f35"}
Oct 30 17:38:46.373245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:46.373212    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" path="/var/lib/kubelet/pods/1444c317-574f-48cd-a200-06c343b2d5ce/volumes"
Oct 30 17:38:47.426867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.426817    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:47.457745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.457715    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:47.535718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.534978    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk\" (UniqueName: \"kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk\") pod \"635e8b8c-0323-49ee-95c5-204b7baa0920\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") "
Oct 30 17:38:47.535718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.535112    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca\") pod \"635e8b8c-0323-49ee-95c5-204b7baa0920\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") "
Oct 30 17:38:47.535718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.535316    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca\") pod \"635e8b8c-0323-49ee-95c5-204b7baa0920\" (UID: \"635e8b8c-0323-49ee-95c5-204b7baa0920\") "
Oct 30 17:38:47.545935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.536609    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca" pod "635e8b8c-0323-49ee-95c5-204b7baa0920" (UID: "635e8b8c-0323-49ee-95c5-204b7baa0920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:47.545935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.539965    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk" pod "635e8b8c-0323-49ee-95c5-204b7baa0920" (UID: "635e8b8c-0323-49ee-95c5-204b7baa0920"). InnerVolumeSpecName "kube-api-access-j6trk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:47.546253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.546217    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca" pod "635e8b8c-0323-49ee-95c5-204b7baa0920" (UID: "635e8b8c-0323-49ee-95c5-204b7baa0920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:38:47.638570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.638534    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6trk\" (UniqueName: \"kubernetes.io/projected/635e8b8c-0323-49ee-95c5-204b7baa0920-kube-api-access-j6trk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:47.638570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.638569    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:47.638812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:47.638583    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635e8b8c-0323-49ee-95c5-204b7baa0920-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:38:48.282729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:48.282696    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"
Oct 30 17:38:48.282919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:48.282701    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg" event={"ID":"635e8b8c-0323-49ee-95c5-204b7baa0920","Type":"ContainerDied","Data":"3edbd3709ffa37131620dba6d07a63ee9e73aee5220ef0290f04a6daf6de8b3f"}
Oct 30 17:38:48.282919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:48.282843    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edbd3709ffa37131620dba6d07a63ee9e73aee5220ef0290f04a6daf6de8b3f"
Oct 30 17:38:48.368204 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:38:48.368171    5534 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635e8b8c_0323_49ee_95c5_204b7baa0920.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 17:38:49.185860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:49.185784    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:50.904963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:50.904923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:52.681082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:52.681045    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:54.173714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:54.173681    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:54.745680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:54.745614    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:55.647115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.647072    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"]
Oct 30 17:38:55.648844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.648816    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy"
Oct 30 17:38:55.648844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.648844    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy"
Oct 30 17:38:55.649038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.648890    5534 cpu_manager.go:414] "containerMap: removing container" podUID="635e8b8c-0323-49ee-95c5-204b7baa0920" containerName="revoke-keys"
Oct 30 17:38:55.649038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.648899    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635e8b8c-0323-49ee-95c5-204b7baa0920" containerName="revoke-keys"
Oct 30 17:38:55.649391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.649371    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1444c317-574f-48cd-a200-06c343b2d5ce" containerName="istio-proxy"
Oct 30 17:38:55.649469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.649403    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="635e8b8c-0323-49ee-95c5-204b7baa0920" containerName="revoke-keys"
Oct 30 17:38:55.662429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.662398    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.674001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.673973    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"]
Oct 30 17:38:55.683385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.683358    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:55.684048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.684027    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:38:55.696159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.696130    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:38:55.709716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.709677    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p28l6\" (UniqueName: \"kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.709906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.709869    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.710066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.710043    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.812321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.812288    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.812494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.812399    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.812555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.812543    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p28l6\" (UniqueName: \"kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.814212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.814179    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.815852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.815826    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.835996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.835969    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p28l6\" (UniqueName: \"kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6\") pod \"maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:55.989771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:55.989689    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:38:56.308164 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:38:56.308133    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e885da_4144_4077_989c_a1c211779d2d.slice/crio-77c0de2004c8b9e45f09ea655fa52b9a4fe88dd9359d9888e5340e560b7c8e7c WatchSource:0}: Error finding container 77c0de2004c8b9e45f09ea655fa52b9a4fe88dd9359d9888e5340e560b7c8e7c: Status 404 returned error can't find the container with id 77c0de2004c8b9e45f09ea655fa52b9a4fe88dd9359d9888e5340e560b7c8e7c
Oct 30 17:38:56.308303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:56.308193    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"]
Oct 30 17:38:56.413405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:56.413360    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4" event={"ID":"a9e885da-4144-4077-989c-a1c211779d2d","Type":"ContainerStarted","Data":"77c0de2004c8b9e45f09ea655fa52b9a4fe88dd9359d9888e5340e560b7c8e7c"}
Oct 30 17:38:56.676842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:56.676813    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:56.799771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:56.799727    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:57.004057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:57.003982    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:57.042705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:57.042676    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"]
Oct 30 17:38:57.064080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:57.064044    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b029ea6-k2hw6"]
Oct 30 17:38:57.423926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:57.423895    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4" event={"ID":"a9e885da-4144-4077-989c-a1c211779d2d","Type":"ContainerStarted","Data":"eb60b4a749f60d6d6e688d1f9585fd23a175cdd327f56eaf474d8698fd181392"}
Oct 30 17:38:57.467734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:57.467672    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4" podStartSLOduration=2.46763585 podStartE2EDuration="2.46763585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:57.467018536 +0000 UTC m=+1543.229062349" watchObservedRunningTime="2026-10-30 17:38:57.46763585 +0000 UTC m=+1543.229679664"
Oct 30 17:38:57.946888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:57.946838    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:58.172697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.168726    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"]
Oct 30 17:38:58.191670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.190948    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.204191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.202049    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"]
Oct 30 17:38:58.280712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.268983    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.280712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.269130    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrk4j\" (UniqueName: \"kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.280712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.269274    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.373812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.372886    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.373812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.373032    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrk4j\" (UniqueName: \"kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.373812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.373170    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.403295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.393074    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.403295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.399180    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.434019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.430951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrk4j\" (UniqueName: \"kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j\") pod \"maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.462517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.462437    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c64afe-863e-477d-8c56-305c6659ea04" path="/var/lib/kubelet/pods/28c64afe-863e-477d-8c56-305c6659ea04/volumes"
Oct 30 17:38:58.633665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.596043    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:38:58.633665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.633375    5534 generic.go:356] "Generic (PLEG): container finished" podID="a9e885da-4144-4077-989c-a1c211779d2d" containerID="eb60b4a749f60d6d6e688d1f9585fd23a175cdd327f56eaf474d8698fd181392" exitCode=0
Oct 30 17:38:58.633928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.633770    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4" event={"ID":"a9e885da-4144-4077-989c-a1c211779d2d","Type":"ContainerDied","Data":"eb60b4a749f60d6d6e688d1f9585fd23a175cdd327f56eaf474d8698fd181392"}
Oct 30 17:38:58.894957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:58.888293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:59.216629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:59.216594    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:59.327508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:59.327471    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"]
Oct 30 17:38:59.332268 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:38:59.332241    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f70b26_a623_43d9_8d69_1891cafd0a0b.slice/crio-eb395b2a3878e0b8f73da92586f1b13d17a74199fb7ad78d9df7dfd57f27cee7 WatchSource:0}: Error finding container eb395b2a3878e0b8f73da92586f1b13d17a74199fb7ad78d9df7dfd57f27cee7: Status 404 returned error can't find the container with id eb395b2a3878e0b8f73da92586f1b13d17a74199fb7ad78d9df7dfd57f27cee7
Oct 30 17:38:59.346451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:59.346418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:38:59.650617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:59.650456    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz" event={"ID":"64f70b26-a623-43d9-8d69-1891cafd0a0b","Type":"ContainerStarted","Data":"eb395b2a3878e0b8f73da92586f1b13d17a74199fb7ad78d9df7dfd57f27cee7"}
Oct 30 17:38:59.952439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:38:59.865482    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:00.056047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.056019    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:39:00.176898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.176863    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca\") pod \"a9e885da-4144-4077-989c-a1c211779d2d\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") "
Oct 30 17:39:00.177124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.177102    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca\") pod \"a9e885da-4144-4077-989c-a1c211779d2d\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") "
Oct 30 17:39:00.177312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.177296    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6\" (UniqueName: \"kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6\") pod \"a9e885da-4144-4077-989c-a1c211779d2d\" (UID: \"a9e885da-4144-4077-989c-a1c211779d2d\") "
Oct 30 17:39:00.188365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.187780    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca" pod "a9e885da-4144-4077-989c-a1c211779d2d" (UID: "a9e885da-4144-4077-989c-a1c211779d2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:00.188365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.187931    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6" pod "a9e885da-4144-4077-989c-a1c211779d2d" (UID: "a9e885da-4144-4077-989c-a1c211779d2d"). InnerVolumeSpecName "kube-api-access-p28l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:00.193253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.188812    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca" pod "a9e885da-4144-4077-989c-a1c211779d2d" (UID: "a9e885da-4144-4077-989c-a1c211779d2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:00.279553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.279472    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:00.279553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.279506    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p28l6\" (UniqueName: \"kubernetes.io/projected/a9e885da-4144-4077-989c-a1c211779d2d-kube-api-access-p28l6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:00.279553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.279522    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e885da-4144-4077-989c-a1c211779d2d-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:00.384505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.384472    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:00.646135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.589382    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:00.726754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.702738    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4" event={"ID":"a9e885da-4144-4077-989c-a1c211779d2d","Type":"ContainerDied","Data":"77c0de2004c8b9e45f09ea655fa52b9a4fe88dd9359d9888e5340e560b7c8e7c"}
Oct 30 17:39:00.726754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.702789    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c0de2004c8b9e45f09ea655fa52b9a4fe88dd9359d9888e5340e560b7c8e7c"
Oct 30 17:39:00.726754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.702849    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"
Oct 30 17:39:00.726754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.706039    5534 generic.go:356] "Generic (PLEG): container finished" podID="64f70b26-a623-43d9-8d69-1891cafd0a0b" containerID="79955b7c1e6b3f68b7e7523c761a8979ce32e51bc6847f9fcbde00b8982299fd" exitCode=0
Oct 30 17:39:00.726754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:00.706144    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz" event={"ID":"64f70b26-a623-43d9-8d69-1891cafd0a0b","Type":"ContainerDied","Data":"79955b7c1e6b3f68b7e7523c761a8979ce32e51bc6847f9fcbde00b8982299fd"}
Oct 30 17:39:01.671468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:01.671237    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:02.064669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.063887    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:39:02.191857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.191822    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca\") pod \"64f70b26-a623-43d9-8d69-1891cafd0a0b\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") "
Oct 30 17:39:02.215997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.214631    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j\" (UniqueName: \"kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j\") pod \"64f70b26-a623-43d9-8d69-1891cafd0a0b\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") "
Oct 30 17:39:02.215997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.214871    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca\") pod \"64f70b26-a623-43d9-8d69-1891cafd0a0b\" (UID: \"64f70b26-a623-43d9-8d69-1891cafd0a0b\") "
Oct 30 17:39:02.216251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.214491    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca" pod "64f70b26-a623-43d9-8d69-1891cafd0a0b" (UID: "64f70b26-a623-43d9-8d69-1891cafd0a0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:02.278822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.268810    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:02.325668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.299456    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca" pod "64f70b26-a623-43d9-8d69-1891cafd0a0b" (UID: "64f70b26-a623-43d9-8d69-1891cafd0a0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:02.325668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.299624    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j" pod "64f70b26-a623-43d9-8d69-1891cafd0a0b" (UID: "64f70b26-a623-43d9-8d69-1891cafd0a0b"). InnerVolumeSpecName "kube-api-access-rrk4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:02.342696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.332750    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:02.342696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.332793    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f70b26-a623-43d9-8d69-1891cafd0a0b-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:02.342696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.332810    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrk4j\" (UniqueName: \"kubernetes.io/projected/64f70b26-a623-43d9-8d69-1891cafd0a0b-kube-api-access-rrk4j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:02.757668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.754915    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz" event={"ID":"64f70b26-a623-43d9-8d69-1891cafd0a0b","Type":"ContainerDied","Data":"eb395b2a3878e0b8f73da92586f1b13d17a74199fb7ad78d9df7dfd57f27cee7"}
Oct 30 17:39:02.757668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.755001    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb395b2a3878e0b8f73da92586f1b13d17a74199fb7ad78d9df7dfd57f27cee7"
Oct 30 17:39:02.757668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:02.755087    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"
Oct 30 17:39:03.399704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:03.391783    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:39:03.399704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:03.391858    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:39:04.012339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:04.012205    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:04.286629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:04.286533    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:04.438927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:04.438888    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:04.940809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:04.940770    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:05.071570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:05.071521    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:05.830556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:05.830484    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:05.962011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:05.961977    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:06.396744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:06.396707    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"]
Oct 30 17:39:06.518915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:06.518876    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:06.652169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:06.652051    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:06.780999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:06.780939    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:08.397237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.397188    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:08.481313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.481278    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"]
Oct 30 17:39:08.481851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.481814    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" podUID="d08d4ddd-ba79-4fe3-899a-619e7b921548" containerName="maas-api" containerID="cri-o://50c9df0ccfce8735c1dd6b0371b8c074d2b9ae7a8f5546d23e26c370625c975b" gracePeriod=30
Oct 30 17:39:08.532630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.532530    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:08.887662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.887603    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:08.970525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.970068    5534 generic.go:356] "Generic (PLEG): container finished" podID="d08d4ddd-ba79-4fe3-899a-619e7b921548" containerID="50c9df0ccfce8735c1dd6b0371b8c074d2b9ae7a8f5546d23e26c370625c975b" exitCode=0
Oct 30 17:39:08.970525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:08.970239    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" event={"ID":"d08d4ddd-ba79-4fe3-899a-619e7b921548","Type":"ContainerDied","Data":"50c9df0ccfce8735c1dd6b0371b8c074d2b9ae7a8f5546d23e26c370625c975b"}
Oct 30 17:39:09.064388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.064351    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"]
Oct 30 17:39:09.128475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.128352    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:09.274215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.267205    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:09.324699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.311359    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:39:09.417095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.415691    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm\" (UniqueName: \"kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm\") pod \"d08d4ddd-ba79-4fe3-899a-619e7b921548\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") "
Oct 30 17:39:09.417095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.415826    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca\") pod \"d08d4ddd-ba79-4fe3-899a-619e7b921548\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") "
Oct 30 17:39:09.417095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.415868    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\") pod \"d08d4ddd-ba79-4fe3-899a-619e7b921548\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") "
Oct 30 17:39:09.418165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.418134    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca" pod "d08d4ddd-ba79-4fe3-899a-619e7b921548" (UID: "d08d4ddd-ba79-4fe3-899a-619e7b921548"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:09.418268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.418224    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs\") pod \"d08d4ddd-ba79-4fe3-899a-619e7b921548\" (UID: \"d08d4ddd-ba79-4fe3-899a-619e7b921548\") "
Oct 30 17:39:09.421817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.421790    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d4ddd-ba79-4fe3-899a-619e7b921548-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:09.423324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.423299    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs" pod "d08d4ddd-ba79-4fe3-899a-619e7b921548" (UID: "d08d4ddd-ba79-4fe3-899a-619e7b921548"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:09.423996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.423962    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls" pod "d08d4ddd-ba79-4fe3-899a-619e7b921548" (UID: "d08d4ddd-ba79-4fe3-899a-619e7b921548"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:09.426175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.426134    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm" pod "d08d4ddd-ba79-4fe3-899a-619e7b921548" (UID: "d08d4ddd-ba79-4fe3-899a-619e7b921548"). InnerVolumeSpecName "kube-api-access-c4xdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:09.493264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.492679    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"]
Oct 30 17:39:09.493444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.493351    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy" containerID="cri-o://85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c" gracePeriod=30
Oct 30 17:39:09.503665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.499161    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:09.524719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.523297    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:09.524719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.523331    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xdm\" (UniqueName: \"kubernetes.io/projected/d08d4ddd-ba79-4fe3-899a-619e7b921548-kube-api-access-c4xdm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:09.524719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.523346    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d4ddd-ba79-4fe3-899a-619e7b921548-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:09.528096 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.528057    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"]
Oct 30 17:39:09.528323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.528294    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy" containerID="cri-o://6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d" gracePeriod=30
Oct 30 17:39:09.535236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.535195    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"]
Oct 30 17:39:09.535477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.535451    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy" containerID="cri-o://ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f" gracePeriod=30
Oct 30 17:39:09.545748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.544051    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"]
Oct 30 17:39:09.545748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.544405    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" containerID="cri-o://a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce" gracePeriod=30
Oct 30 17:39:09.906671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:09.905518    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:10.008048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.005945    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"]
Oct 30 17:39:10.008048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.006010    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7bb2e69d-x7rs4"]
Oct 30 17:39:10.032662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.028510    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:10.040799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.040760    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm" event={"ID":"d08d4ddd-ba79-4fe3-899a-619e7b921548","Type":"ContainerDied","Data":"0571b222e4f930fbe35f8b10c43bfedd90858b944a895121c01bcb6cf6820c14"}
Oct 30 17:39:10.040981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.040896    5534 scope.go:122] "RemoveContainer" containerID="50c9df0ccfce8735c1dd6b0371b8c074d2b9ae7a8f5546d23e26c370625c975b"
Oct 30 17:39:10.041125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.041101    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"
Oct 30 17:39:10.128056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.128026    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"]
Oct 30 17:39:10.131894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.131866    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7bb2e69d-6b75f4b5d-5hbvm"]
Oct 30 17:39:10.376921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.376885    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e885da-4144-4077-989c-a1c211779d2d" path="/var/lib/kubelet/pods/a9e885da-4144-4077-989c-a1c211779d2d/volumes"
Oct 30 17:39:10.377571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:10.377535    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08d4ddd-ba79-4fe3-899a-619e7b921548" path="/var/lib/kubelet/pods/d08d4ddd-ba79-4fe3-899a-619e7b921548/volumes"
Oct 30 17:39:11.398181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.398128    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59" podUID="d488f290-f98c-42b7-b0f5-572d1dc96d34" containerName="payload-processing" containerID="cri-o://2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248" gracePeriod=25
Oct 30 17:39:11.771720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.760176    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:39:11.908486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.908456    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68\" (UniqueName: \"kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68\") pod \"d488f290-f98c-42b7-b0f5-572d1dc96d34\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") "
Oct 30 17:39:11.908685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.908557    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume\") pod \"d488f290-f98c-42b7-b0f5-572d1dc96d34\" (UID: \"d488f290-f98c-42b7-b0f5-572d1dc96d34\") "
Oct 30 17:39:11.908982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.908960    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume" pod "d488f290-f98c-42b7-b0f5-572d1dc96d34" (UID: "d488f290-f98c-42b7-b0f5-572d1dc96d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:11.909094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.909079    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d488f290-f98c-42b7-b0f5-572d1dc96d34-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:11.910875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:11.910851    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68" pod "d488f290-f98c-42b7-b0f5-572d1dc96d34" (UID: "d488f290-f98c-42b7-b0f5-572d1dc96d34"). InnerVolumeSpecName "kube-api-access-txv68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:12.010924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.010838    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txv68\" (UniqueName: \"kubernetes.io/projected/d488f290-f98c-42b7-b0f5-572d1dc96d34-kube-api-access-txv68\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:12.061609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.061572    5534 generic.go:356] "Generic (PLEG): container finished" podID="d488f290-f98c-42b7-b0f5-572d1dc96d34" containerID="2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248" exitCode=0
Oct 30 17:39:12.061815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.061632    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"
Oct 30 17:39:12.061815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.061673    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59" event={"ID":"d488f290-f98c-42b7-b0f5-572d1dc96d34","Type":"ContainerDied","Data":"2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248"}
Oct 30 17:39:12.061815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.061744    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59" event={"ID":"d488f290-f98c-42b7-b0f5-572d1dc96d34","Type":"ContainerDied","Data":"18fcbbd3d6dc36bec876ec71648334a3ea7e24351e3feea7c4bd16ee72414d5b"}
Oct 30 17:39:12.061815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.061768    5534 scope.go:122] "RemoveContainer" containerID="2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248"
Oct 30 17:39:12.074332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.074293    5534 scope.go:122] "RemoveContainer" containerID="2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248"
Oct 30 17:39:12.074796 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:12.074767    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248\": container with ID starting with 2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248 not found: ID does not exist" containerID="2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248"
Oct 30 17:39:12.074923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.074820    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248"} err="failed to get container status \"2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248\": rpc error: code = NotFound desc = could not find container \"2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248\": container with ID starting with 2d4c1828c056f6b7a662add7b2ed9b852ae6b51f3426216e9f7f53670e1cd248 not found: ID does not exist"
Oct 30 17:39:12.114990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.114958    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"]
Oct 30 17:39:12.118771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.118745    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7bb2e69d-68d847746-llq59"]
Oct 30 17:39:12.158730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.158692    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:12.372433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:12.372399    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d488f290-f98c-42b7-b0f5-572d1dc96d34" path="/var/lib/kubelet/pods/d488f290-f98c-42b7-b0f5-572d1dc96d34/volumes"
Oct 30 17:39:13.916155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:13.916117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:14.065967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.065925    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq" podUID="fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" containerName="payload-processing" containerID="cri-o://a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e" gracePeriod=25
Oct 30 17:39:14.366554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.366527    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:39:14.387690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.387660    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume\") pod \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") "
Oct 30 17:39:14.387878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.387847    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt\" (UniqueName: \"kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt\") pod \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\" (UID: \"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7\") "
Oct 30 17:39:14.387993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.387967    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume" pod "fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" (UID: "fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:14.388450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.388431    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:14.390080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.390050    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt" pod "fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" (UID: "fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7"). InnerVolumeSpecName "kube-api-access-ts5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:14.490022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.489948    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5wt\" (UniqueName: \"kubernetes.io/projected/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7-kube-api-access-ts5wt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:14.856171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.856135    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.921688    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.921954    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.921997    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.922047    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.922098    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.922215    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.922272    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.922301    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.922432    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket\") pod \"5cf6850e-ee80-499e-98c8-346402e7b283\" (UID: \"5cf6850e-ee80-499e-98c8-346402e7b283\") "
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.923707    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.923966    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:14.927709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.927583    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:14.928810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.927916    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:14.928810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.928161    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:14.928810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.928753    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:39:14.952002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.930400    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:14.952002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.930477    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:14.970117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.969862    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr" pod "5cf6850e-ee80-499e-98c8-346402e7b283" (UID: "5cf6850e-ee80-499e-98c8-346402e7b283"). InnerVolumeSpecName "kube-api-access-6rkdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:14.973390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.973318    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:39:14.992099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.991940    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:39:14.999834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:14.999757    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:39:15.024748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.024676    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.028547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.028235    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.028547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.028330    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.028547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.028424    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.033267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.028885    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.033267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.028983    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.033267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.029060    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.033267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.029100    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.034247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.033811    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data\") pod \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\" (UID: \"e9c282ed-89ac-48f6-9f4e-7946576d5dd0\") "
Oct 30 17:39:15.035060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.035035    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038582    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038609    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf6850e-ee80-499e-98c8-346402e7b283-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038634    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038668    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf6850e-ee80-499e-98c8-346402e7b283-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038683    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038694    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038707    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf6850e-ee80-499e-98c8-346402e7b283-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.038957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.038720    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkdr\" (UniqueName: \"kubernetes.io/projected/5cf6850e-ee80-499e-98c8-346402e7b283-kube-api-access-6rkdr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.039786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.039458    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.049481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.049203    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.051505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.051476    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:15.051877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.051754    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.052084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.052053    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.054098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.054058    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:39:15.054610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.054584    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.055130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.055102    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "kube-api-access-vdfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:15.056456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.056422    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token" pod "e9c282ed-89ac-48f6-9f4e-7946576d5dd0" (UID: "e9c282ed-89ac-48f6-9f4e-7946576d5dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:15.118782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.118713    5534 generic.go:356] "Generic (PLEG): container finished" podID="5cf6850e-ee80-499e-98c8-346402e7b283" containerID="85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c" exitCode=0
Oct 30 17:39:15.118956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.118874    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" event={"ID":"5cf6850e-ee80-499e-98c8-346402e7b283","Type":"ContainerDied","Data":"85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c"}
Oct 30 17:39:15.119035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.118958    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t" event={"ID":"5cf6850e-ee80-499e-98c8-346402e7b283","Type":"ContainerDied","Data":"e490eb81d83bd4ddda263e1f87f208880c90673d650d1c5d66f44beb49f0faf0"}
Oct 30 17:39:15.119035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.119021    5534 scope.go:122] "RemoveContainer" containerID="85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c"
Oct 30 17:39:15.119269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.119250    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"
Oct 30 17:39:15.135319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.135283    5534 generic.go:356] "Generic (PLEG): container finished" podID="fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" containerID="a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e" exitCode=0
Oct 30 17:39:15.135466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.135444    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq" event={"ID":"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7","Type":"ContainerDied","Data":"a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e"}
Oct 30 17:39:15.135536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.135520    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq" event={"ID":"fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7","Type":"ContainerDied","Data":"9911fcbaa064ccdc5dcd104ed4e5b7b4a2dbda1d4a723c6ff35840376b165d86"}
Oct 30 17:39:15.136596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.135666    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"
Oct 30 17:39:15.143611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.143576    5534 generic.go:356] "Generic (PLEG): container finished" podID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerID="a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce" exitCode=0
Oct 30 17:39:15.146479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.146448    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" event={"ID":"3f21b507-00b3-4988-b1b6-1dabe8f90ca9","Type":"ContainerDied","Data":"a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce"}
Oct 30 17:39:15.146794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.146513    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" event={"ID":"3f21b507-00b3-4988-b1b6-1dabe8f90ca9","Type":"ContainerDied","Data":"90252ba551fc6977dee98f9761509dcdb2322423c58e2d1ff135eb6c04d233fb"}
Oct 30 17:39:15.146794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.146611    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"
Oct 30 17:39:15.150511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.150482    5534 generic.go:356] "Generic (PLEG): container finished" podID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerID="ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f" exitCode=0
Oct 30 17:39:15.150849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.150827    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" event={"ID":"1f1c85fd-404c-4fd6-b059-797f057acf02","Type":"ContainerDied","Data":"ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f"}
Oct 30 17:39:15.150987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.150971    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr" event={"ID":"1f1c85fd-404c-4fd6-b059-797f057acf02","Type":"ContainerDied","Data":"f1efbb574c485c2ed2c4ef351f55273716c93fa4401238d0180767ae660abad7"}
Oct 30 17:39:15.151628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.151602    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.151847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.151830    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.151965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.151953    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.152079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152066    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.152179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152167    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.152361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152347    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.152479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152465    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.152591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152579    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.152756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152741    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.152905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.152891    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.153027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153014    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.153150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153138    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.153255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153243    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.153413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153396    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.153540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153527    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.153725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153711    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket\") pod \"1f1c85fd-404c-4fd6-b059-797f057acf02\" (UID: \"1f1c85fd-404c-4fd6-b059-797f057acf02\") "
Oct 30 17:39:15.153841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153828    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.153982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.153969    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy\") pod \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\" (UID: \"3f21b507-00b3-4988-b1b6-1dabe8f90ca9\") "
Oct 30 17:39:15.154865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.154845    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.154977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.154966    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155052    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155133    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfrz\" (UniqueName: \"kubernetes.io/projected/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-kube-api-access-vdfrz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155215    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155297    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155372    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155493    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.155595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155585    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c282ed-89ac-48f6-9f4e-7946576d5dd0-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.155146    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.161123    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.161374    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.161595    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.161722    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.161828    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.162237    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.162420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.162252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.163180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.163136    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.163471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.163375    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.164018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.163928    5534 scope.go:122] "RemoveContainer" containerID="85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c"
Oct 30 17:39:15.164018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.163971    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"
Oct 30 17:39:15.164541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.164511    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:15.166463 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:15.166386    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c\": container with ID starting with 85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c not found: ID does not exist" containerID="85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c"
Oct 30 17:39:15.166463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.166422    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c"} err="failed to get container status \"85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c\": rpc error: code = NotFound desc = could not find container \"85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c\": container with ID starting with 85e14f7bc73a6b3ab51af9b8ffd8c303a23454ff2fb2c7fec39be5b92c97d71c not found: ID does not exist"
Oct 30 17:39:15.166463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.166446    5534 scope.go:122] "RemoveContainer" containerID="a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e"
Oct 30 17:39:15.167546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.167184    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.169810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.169483    5534 generic.go:356] "Generic (PLEG): container finished" podID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerID="6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d" exitCode=0
Oct 30 17:39:15.169810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.169538    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" event={"ID":"e9c282ed-89ac-48f6-9f4e-7946576d5dd0","Type":"ContainerDied","Data":"6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d"}
Oct 30 17:39:15.169810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.169582    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng" event={"ID":"e9c282ed-89ac-48f6-9f4e-7946576d5dd0","Type":"ContainerDied","Data":"22b196ece042291927ff789974389a0eb59a03c5dfc952a3f1819ea5fd23ce4b"}
Oct 30 17:39:15.170690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.170068    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"
Oct 30 17:39:15.172802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.172763    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:39:15.177081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.176783    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "kube-api-access-7nvwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:15.190094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.189052    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:15.190094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.189454    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo" pod "3f21b507-00b3-4988-b1b6-1dabe8f90ca9" (UID: "3f21b507-00b3-4988-b1b6-1dabe8f90ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:39:15.190094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.189873    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:15.190471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.190353    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5" pod "1f1c85fd-404c-4fd6-b059-797f057acf02" (UID: "1f1c85fd-404c-4fd6-b059-797f057acf02"). InnerVolumeSpecName "kube-api-access-cs5g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:15.237448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.236832    5534 scope.go:122] "RemoveContainer" containerID="a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e"
Oct 30 17:39:15.239161 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:15.238398    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e\": container with ID starting with a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e not found: ID does not exist" containerID="a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e"
Oct 30 17:39:15.239161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.238440    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e"} err="failed to get container status \"a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e\": rpc error: code = NotFound desc = could not find container \"a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e\": container with ID starting with a559dfc918ec18d1feabfc31b333b3b053fa0afbcbcfc4687bef4290a035eb4e not found: ID does not exist"
Oct 30 17:39:15.239161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.238468    5534 scope.go:122] "RemoveContainer" containerID="a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce"
Oct 30 17:39:15.240749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.240722    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"]
Oct 30 17:39:15.245918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.245895    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-5469t"]
Oct 30 17:39:15.262139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.262117    5534 scope.go:122] "RemoveContainer" containerID="a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce"
Oct 30 17:39:15.262989 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:15.262819    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce\": container with ID starting with a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce not found: ID does not exist" containerID="a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce"
Oct 30 17:39:15.262989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.262862    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce"} err="failed to get container status \"a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce\": rpc error: code = NotFound desc = could not find container \"a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce\": container with ID starting with a0a88d20df44440a86a97f44c6e7c609b5f0468bc8cd4ce748d0912747f35dce not found: ID does not exist"
Oct 30 17:39:15.262989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.262886    5534 scope.go:122] "RemoveContainer" containerID="ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f"
Oct 30 17:39:15.265316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.265292    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nvwq\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-kube-api-access-7nvwq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.265509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.265493    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.265607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.265597    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.265794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.265781    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.265887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.265877    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.265966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.265957    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266035    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266125    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5g5\" (UniqueName: \"kubernetes.io/projected/1f1c85fd-404c-4fd6-b059-797f057acf02-kube-api-access-cs5g5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266211    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266296    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266379    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266462    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266539    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266613    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266738    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1c85fd-404c-4fd6-b059-797f057acf02-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266828    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1c85fd-404c-4fd6-b059-797f057acf02-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266902    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1c85fd-404c-4fd6-b059-797f057acf02-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.266977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.266968    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f21b507-00b3-4988-b1b6-1dabe8f90ca9-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:15.278668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.278546    5534 scope.go:122] "RemoveContainer" containerID="ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f"
Oct 30 17:39:15.279097 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:15.278949    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f\": container with ID starting with ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f not found: ID does not exist" containerID="ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f"
Oct 30 17:39:15.279097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.278987    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f"} err="failed to get container status \"ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f\": rpc error: code = NotFound desc = could not find container \"ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f\": container with ID starting with ce3fa5a4d673e669db7b5634721f14319ce10445b77c9a820662e7a9763c666f not found: ID does not exist"
Oct 30 17:39:15.279097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.279010    5534 scope.go:122] "RemoveContainer" containerID="6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d"
Oct 30 17:39:15.286583 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:15.286556    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:39:15.293694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.293503    5534 scope.go:122] "RemoveContainer" containerID="6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d"
Oct 30 17:39:15.294062 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:15.294027    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d\": container with ID starting with 6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d not found: ID does not exist" containerID="6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d"
Oct 30 17:39:15.294324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.294066    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d"} err="failed to get container status \"6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d\": rpc error: code = NotFound desc = could not find container \"6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d\": container with ID starting with 6718b961c64c9fd6d8ad8e7bae08f2f54def6d14355938a524d9dcf6d5ea058d not found: ID does not exist"
Oct 30 17:39:15.301906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.301841    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"]
Oct 30 17:39:15.305860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.302138    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" podUID="a777fcc3-983e-4a83-9de3-29dc81ec1cc2" containerName="maas-api" containerID="cri-o://023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a" gracePeriod=30
Oct 30 17:39:15.322808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.322736    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"]
Oct 30 17:39:15.329872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.329838    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-xr4ng"]
Oct 30 17:39:15.377342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.377295    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:15.381717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.381696    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"]
Oct 30 17:39:15.395519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.395493    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df8b20-b98d88686-sqntq"]
Oct 30 17:39:15.506531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.506496    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"]
Oct 30 17:39:15.515517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.515435    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk"]
Oct 30 17:39:15.533315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.533274    5534 scope.go:122] "RemoveContainer" containerID="59093c98acfb2bb83aa04ea5d747e608aea4e09c97e91850d76a7fcfad2612fa"
Oct 30 17:39:15.805590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.805549    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"]
Oct 30 17:39:15.831206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.831176    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:39:15.857998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.857845    5534 scope.go:122] "RemoveContainer" containerID="62b62d4aadd8aab2b8dcb01838e3bbec8f4cf290d55329ea76eb3cb036e8948b"
Oct 30 17:39:15.929422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.928995    5534 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:39:15.929422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.929064    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-zhxkk" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:39:15.963349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:15.963316    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:16.014982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.014946    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\") pod \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") "
Oct 30 17:39:16.015259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.015242    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l\" (UniqueName: \"kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l\") pod \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") "
Oct 30 17:39:16.019011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.018945    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca\") pod \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") "
Oct 30 17:39:16.019375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.019356    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs\") pod \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\" (UID: \"a777fcc3-983e-4a83-9de3-29dc81ec1cc2\") "
Oct 30 17:39:16.023057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.023031    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca" pod "a777fcc3-983e-4a83-9de3-29dc81ec1cc2" (UID: "a777fcc3-983e-4a83-9de3-29dc81ec1cc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:16.040472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.040403    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-b4mjr"]
Oct 30 17:39:16.055925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.055632    5534 scope.go:122] "RemoveContainer" containerID="8759174a0a7fefa7129cd92c094bb25fd1c20331498cdd054c3d402241a872f8"
Oct 30 17:39:16.061007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.060978    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs" pod "a777fcc3-983e-4a83-9de3-29dc81ec1cc2" (UID: "a777fcc3-983e-4a83-9de3-29dc81ec1cc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:16.062533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.062446    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls" pod "a777fcc3-983e-4a83-9de3-29dc81ec1cc2" (UID: "a777fcc3-983e-4a83-9de3-29dc81ec1cc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:16.072840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.071924    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l" pod "a777fcc3-983e-4a83-9de3-29dc81ec1cc2" (UID: "a777fcc3-983e-4a83-9de3-29dc81ec1cc2"). InnerVolumeSpecName "kube-api-access-px95l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:16.076007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.075811    5534 scope.go:122] "RemoveContainer" containerID="91b53e7cafb716ef06da2bfdd3d779816d09ea80e9b4f66d3075f51950341033"
Oct 30 17:39:16.109983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.109957    5534 scope.go:122] "RemoveContainer" containerID="c72a079a6c2547475297ce3ae80e7193c7da8e40b830432e8f9b9dcc927a90f9"
Oct 30 17:39:16.120997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.120884    5534 scope.go:122] "RemoveContainer" containerID="8bbee5bb38d58bdf0e16ee3549a536f4e4215e3d7a109e5b12145f81c0e995af"
Oct 30 17:39:16.122126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.122102    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px95l\" (UniqueName: \"kubernetes.io/projected/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-kube-api-access-px95l\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:16.122207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.122134    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:16.122207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.122150    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:16.122207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.122165    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777fcc3-983e-4a83-9de3-29dc81ec1cc2-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:16.130503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.130477    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:16.132703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.132627    5534 scope.go:122] "RemoveContainer" containerID="e9546421708917ec67ff3a7104099e30dae0bc735ccc34f73cf73a0f60e091a1"
Oct 30 17:39:16.143480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.143460    5534 scope.go:122] "RemoveContainer" containerID="c0fa8c22f75feee05803df0f09c2c15d4ebcaefd23cd1bc9b7469cdc4eb9bdc8"
Oct 30 17:39:16.154866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.154843    5534 scope.go:122] "RemoveContainer" containerID="7836c5932e26002a96239d25dfab346e178043569e5d6c967d42ab13e36a5346"
Oct 30 17:39:16.168235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.168195    5534 scope.go:122] "RemoveContainer" containerID="b77778a8c459e2333128cc394c7cab4904d7e9e6b646b1b7fd5db8e6b4cece2e"
Oct 30 17:39:16.201840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.201816    5534 scope.go:122] "RemoveContainer" containerID="e3a7136ac6029147536fffc837527b2952fd29f7c189eba61a46f538e543d1c3"
Oct 30 17:39:16.225697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.223155    5534 scope.go:122] "RemoveContainer" containerID="564d65f49947ccdf76e3ce24fba84759fc3c18ac231b013072193cae9c05be1b"
Oct 30 17:39:16.226942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.226912    5534 generic.go:356] "Generic (PLEG): container finished" podID="a777fcc3-983e-4a83-9de3-29dc81ec1cc2" containerID="023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a" exitCode=0
Oct 30 17:39:16.227251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.227094    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" event={"ID":"a777fcc3-983e-4a83-9de3-29dc81ec1cc2","Type":"ContainerDied","Data":"023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a"}
Oct 30 17:39:16.227458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.227442    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g" event={"ID":"a777fcc3-983e-4a83-9de3-29dc81ec1cc2","Type":"ContainerDied","Data":"882fe726f12981e9fb7b88f77f814d5f2aefb821181a7d0f6bd3e3ea64cf37e1"}
Oct 30 17:39:16.227597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.227547    5534 scope.go:122] "RemoveContainer" containerID="023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a"
Oct 30 17:39:16.234157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.227150    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"
Oct 30 17:39:16.277344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.277247    5534 scope.go:122] "RemoveContainer" containerID="023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a"
Oct 30 17:39:16.277738 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:16.277561    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a\": container with ID starting with 023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a not found: ID does not exist" containerID="023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a"
Oct 30 17:39:16.277738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.277597    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a"} err="failed to get container status \"023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a\": rpc error: code = NotFound desc = could not find container \"023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a\": container with ID starting with 023d50c01ccd322085074133bed99b01aadf7ee48ccb62cf8c776caba1778b6a not found: ID does not exist"
Oct 30 17:39:16.317170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.317143    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"]
Oct 30 17:39:16.337047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.337019    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-hft9g"]
Oct 30 17:39:16.355662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.350470    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"]
Oct 30 17:39:16.356134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.356103    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-nff5s"]
Oct 30 17:39:16.374469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.374436    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" path="/var/lib/kubelet/pods/1f1c85fd-404c-4fd6-b059-797f057acf02/volumes"
Oct 30 17:39:16.375209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.375183    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" path="/var/lib/kubelet/pods/3f21b507-00b3-4988-b1b6-1dabe8f90ca9/volumes"
Oct 30 17:39:16.375845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.375824    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" path="/var/lib/kubelet/pods/5cf6850e-ee80-499e-98c8-346402e7b283/volumes"
Oct 30 17:39:16.376967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.376946    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61071055-3b08-4cfc-967a-be7b35d88aa1" path="/var/lib/kubelet/pods/61071055-3b08-4cfc-967a-be7b35d88aa1/volumes"
Oct 30 17:39:16.377524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.377501    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a777fcc3-983e-4a83-9de3-29dc81ec1cc2" path="/var/lib/kubelet/pods/a777fcc3-983e-4a83-9de3-29dc81ec1cc2/volumes"
Oct 30 17:39:16.378172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.378152    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" path="/var/lib/kubelet/pods/e9c282ed-89ac-48f6-9f4e-7946576d5dd0/volumes"
Oct 30 17:39:16.379910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.379839    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" path="/var/lib/kubelet/pods/fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7/volumes"
Oct 30 17:39:16.499898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:16.499860    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:18.376310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:18.376278    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:20.001126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:20.001087    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:20.473306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:20.473259    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:21.701951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:21.701910    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:23.481260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:23.481220    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:25.094035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:25.093993    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:26.633386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:26.633340    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:26.749059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:26.749026    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:27.010945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:27.010830    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"]
Oct 30 17:39:27.011138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:27.011104    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" containerID="cri-o://14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470" gracePeriod=30
Oct 30 17:39:27.142689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:27.142654    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:28.114604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:28.114568    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:28.240339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:28.240294    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:28.622716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:28.622685    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:29.031950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:29.031869    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"]
Oct 30 17:39:29.043502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:29.043474    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72dd180-2724r"]
Oct 30 17:39:29.353125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:29.353094    5534 patch_prober.go:29] interesting pod/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:39:29.353548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:29.353155    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:39:30.325933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:30.325896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:30.381443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:30.381404    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ce3963-ebe2-4fbb-a4d6-5befc42b96d0" path="/var/lib/kubelet/pods/08ce3963-ebe2-4fbb-a4d6-5befc42b96d0/volumes"
Oct 30 17:39:30.737467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:30.737425    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:31.922875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:31.922845    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:32.303056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.303030    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:39:32.371657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.371605    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerID="14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470" exitCode=0
Oct 30 17:39:32.371852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.371836    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"
Oct 30 17:39:32.378091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.378060    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" event={"ID":"2f7cc5c1-f70f-4fd6-a347-5b40db00e160","Type":"ContainerDied","Data":"14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470"}
Oct 30 17:39:32.378245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.378124    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79" event={"ID":"2f7cc5c1-f70f-4fd6-a347-5b40db00e160","Type":"ContainerDied","Data":"f9cbbfe8a5e1431bc909cf4df7fe48d4aff71d16182afd7585ffedb948122c97"}
Oct 30 17:39:32.378245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.378153    5534 scope.go:122] "RemoveContainer" containerID="14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470"
Oct 30 17:39:32.393153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.393129    5534 scope.go:122] "RemoveContainer" containerID="b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"
Oct 30 17:39:32.406898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.406866    5534 scope.go:122] "RemoveContainer" containerID="14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470"
Oct 30 17:39:32.407224 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:32.407198    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470\": container with ID starting with 14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470 not found: ID does not exist" containerID="14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470"
Oct 30 17:39:32.407330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.407238    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470"} err="failed to get container status \"14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470\": rpc error: code = NotFound desc = could not find container \"14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470\": container with ID starting with 14a68485641ea8137264a8d55f31fcb3feaaf4f854e095c234a76cee9043d470 not found: ID does not exist"
Oct 30 17:39:32.407330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.407262    5534 scope.go:122] "RemoveContainer" containerID="b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"
Oct 30 17:39:32.407595 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:32.407562    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f\": container with ID starting with b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f not found: ID does not exist" containerID="b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"
Oct 30 17:39:32.407746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.407603    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f"} err="failed to get container status \"b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f\": rpc error: code = NotFound desc = could not find container \"b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f\": container with ID starting with b675400845a4c0fdbd0c7f2b346908290c9e49817e1d23082b6544c4307fcc8f not found: ID does not exist"
Oct 30 17:39:32.430259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430235    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430278    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430359    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430450    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430528    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:32.430609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430567    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430594    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430679    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430721    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430830    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data\") pod \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\" (UID: \"2f7cc5c1-f70f-4fd6-a347-5b40db00e160\") "
Oct 30 17:39:32.430880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.430830    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:32.431159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431107    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:32.431159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431135    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:32.431512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431482    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:32.431625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431567    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.431625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431587    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.431777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431602    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.431777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.431668    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.433325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.433294    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:39:32.433602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.433582    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:32.433796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.433776    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:32.433990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.433971    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn" pod "2f7cc5c1-f70f-4fd6-a347-5b40db00e160" (UID: "2f7cc5c1-f70f-4fd6-a347-5b40db00e160"). InnerVolumeSpecName "kube-api-access-pz7wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:32.534139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.534104    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.534139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.534139    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.534335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.534155    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.534335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.534166    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7wn\" (UniqueName: \"kubernetes.io/projected/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-kube-api-access-pz7wn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.534335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.534174    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7cc5c1-f70f-4fd6-a347-5b40db00e160-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:32.779381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.779338    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"]
Oct 30 17:39:32.790082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:32.790036    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7bb2e69d-openshift-default-794b8d846f-k5z79"]
Oct 30 17:39:33.669479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.669444    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"]
Oct 30 17:39:33.670248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670226    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy"
Oct 30 17:39:33.670248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670246    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670257    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670262    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670281    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" containerName="payload-processing"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670287    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" containerName="payload-processing"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670297    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a9e885da-4144-4077-989c-a1c211779d2d" containerName="revoke-keys"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670302    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e885da-4144-4077-989c-a1c211779d2d" containerName="revoke-keys"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670314    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670319    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670329    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670333    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670341    5534 cpu_manager.go:414] "containerMap: removing container" podUID="64f70b26-a623-43d9-8d69-1891cafd0a0b" containerName="revoke-keys"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670345    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f70b26-a623-43d9-8d69-1891cafd0a0b" containerName="revoke-keys"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670353    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670358    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670369    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d488f290-f98c-42b7-b0f5-572d1dc96d34" containerName="payload-processing"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670374    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d488f290-f98c-42b7-b0f5-572d1dc96d34" containerName="payload-processing"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670381    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d08d4ddd-ba79-4fe3-899a-619e7b921548" containerName="maas-api"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670385    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08d4ddd-ba79-4fe3-899a-619e7b921548" containerName="maas-api"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670394    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a777fcc3-983e-4a83-9de3-29dc81ec1cc2" containerName="maas-api"
Oct 30 17:39:33.670434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670400    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a777fcc3-983e-4a83-9de3-29dc81ec1cc2" containerName="maas-api"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670503    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f70b26-a623-43d9-8d69-1891cafd0a0b" containerName="revoke-keys"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670515    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1c85fd-404c-4fd6-b059-797f057acf02" containerName="istio-proxy"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670523    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d488f290-f98c-42b7-b0f5-572d1dc96d34" containerName="payload-processing"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670529    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08d4ddd-ba79-4fe3-899a-619e7b921548" containerName="maas-api"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670535    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670539    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a777fcc3-983e-4a83-9de3-29dc81ec1cc2" containerName="maas-api"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670549    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f21b507-00b3-4988-b1b6-1dabe8f90ca9" containerName="istio-proxy"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670554    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf6850e-ee80-499e-98c8-346402e7b283" containerName="istio-proxy"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670560    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb355b21-bfc6-4ac2-aeb2-a60ce3b448f7" containerName="payload-processing"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670567    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e885da-4144-4077-989c-a1c211779d2d" containerName="revoke-keys"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670573    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy"
Oct 30 17:39:33.671153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.670579    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c282ed-89ac-48f6-9f4e-7946576d5dd0" containerName="istio-proxy"
Oct 30 17:39:33.674182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.674161    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.696738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.696711    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"]
Oct 30 17:39:33.715080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.713775    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c9bdd9-serving-cert\""
Oct 30 17:39:33.757879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.757844    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.758071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.757915    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:33.758071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.758027    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.758186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.758091    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.758250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.758197    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.859487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.859438    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.859754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.859545    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.859754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.859601    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.859888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.859804    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.860902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.860871    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.863929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.862951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.865485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.865465    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.872099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.872070    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:33.882565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.882544    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2\") pod \"maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:33.996218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.996121    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:33.996474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:33.996453    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:34.278688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:34.265057    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:34.401686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:34.398493    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:34.408389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:34.408255    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" path="/var/lib/kubelet/pods/2f7cc5c1-f70f-4fd6-a347-5b40db00e160/volumes"
Oct 30 17:39:34.490545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:34.490505    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"]
Oct 30 17:39:34.492544 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:34.492515    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf849c571_d2c8_457f_a01b_ebbc08bbe325.slice/crio-de6381da4fe57acb4300274fddf553de2c7b214a671b3ca18955ff6c6fbce3d3 WatchSource:0}: Error finding container de6381da4fe57acb4300274fddf553de2c7b214a671b3ca18955ff6c6fbce3d3: Status 404 returned error can't find the container with id de6381da4fe57acb4300274fddf553de2c7b214a671b3ca18955ff6c6fbce3d3
Oct 30 17:39:34.494526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:34.494506    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:39:35.277735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.274205    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:35.532713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.531714    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"]
Oct 30 17:39:35.536367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.532958    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy"
Oct 30 17:39:35.536367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.532983    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7cc5c1-f70f-4fd6-a347-5b40db00e160" containerName="istio-proxy"
Oct 30 17:39:35.624104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.571470    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:35.633171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.633130    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:35.672703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.648984    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81e9336e\""
Oct 30 17:39:35.672703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.649137    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81e9336e-dockercfg-bcw4c\""
Oct 30 17:39:35.738666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.714101    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plglm\" (UniqueName: \"kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:35.802800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.800826    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:35.802800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.740166    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"]
Oct 30 17:39:35.896246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.895037    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j" event={"ID":"f849c571-d2c8-457f-a01b-ebbc08bbe325","Type":"ContainerStarted","Data":"4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8"}
Oct 30 17:39:35.896246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.895123    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j" event={"ID":"f849c571-d2c8-457f-a01b-ebbc08bbe325","Type":"ContainerStarted","Data":"de6381da4fe57acb4300274fddf553de2c7b214a671b3ca18955ff6c6fbce3d3"}
Oct 30 17:39:35.897976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.897057    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:35.920582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.913831    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plglm\" (UniqueName: \"kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:35.920582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.913935    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:35.920582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:35.915221    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:36.011197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.011164    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"]
Oct 30 17:39:36.018585 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.018560    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.055098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.043633    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plglm\" (UniqueName: \"kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm\") pod \"payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:36.078056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.077298    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"]
Oct 30 17:39:36.129674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.118780    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.129674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.118947    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxmz\" (UniqueName: \"kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.134966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.134659    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j" podStartSLOduration=2.69494883 podStartE2EDuration="3.134610779s" totalImagesPullingTime="439.661949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:36.008729753 +0000 UTC m=+1581.770773590" watchObservedRunningTime="2026-10-30 17:39:36.134610779 +0000 UTC m=+1581.896654594"
Oct 30 17:39:36.193196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.192923    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:36.298668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.274844    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxmz\" (UniqueName: \"kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.298668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.275131    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.298668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.276225    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.364016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.361699    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"]
Oct 30 17:39:36.451474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.451433    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxmz\" (UniqueName: \"kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.509059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.502300    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:36.524788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.524759    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.566683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.546980    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-802bda-serving-cert\""
Oct 30 17:39:36.697690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.691141    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:36.711672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.709815    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:36.748668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.747733    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv9p\" (UniqueName: \"kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.748668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.747821    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.748668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.748193    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.748668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.748259    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.966536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.943454    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.966536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.943756    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.966536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.943840    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.966536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.944045    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv9p\" (UniqueName: \"kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:36.966536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:36.966265    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:37.357133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.345947    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:37.541718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.537479    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:39:37.583691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.558781    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:39:37.588418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.588303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:37.663719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.641178    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:39:37.680064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.675926    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:39:37.743360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.743324    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:37.813053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.744378    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv9p\" (UniqueName: \"kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p\") pod \"maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:37.813053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.745899    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 17:39:37.861620 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:37.840390    5534 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.331s"
Oct 30 17:39:37.861620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.840521    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"]
Oct 30 17:39:37.861620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.840567    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"]
Oct 30 17:39:37.861620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:37.861126    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:38.159866 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:38.159593    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba41d0af_fed9_4bdf_9cdd_9226e61b1385.slice/crio-e614588531f31912a816ff2e2cd4398ea3e4bb181762d90bc6dc674c03f1ef9f WatchSource:0}: Error finding container e614588531f31912a816ff2e2cd4398ea3e4bb181762d90bc6dc674c03f1ef9f: Status 404 returned error can't find the container with id e614588531f31912a816ff2e2cd4398ea3e4bb181762d90bc6dc674c03f1ef9f
Oct 30 17:39:38.162902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.162843    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"]
Oct 30 17:39:38.261514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.261478    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"]
Oct 30 17:39:38.266294 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:38.266267    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d881eb0_5569_439f_859c_a86297985e58.slice/crio-bcec0a1441ff04f35c5951becd669d3dc5ce03437e5f1e283e6e7cd0f1d738fb WatchSource:0}: Error finding container bcec0a1441ff04f35c5951becd669d3dc5ce03437e5f1e283e6e7cd0f1d738fb: Status 404 returned error can't find the container with id bcec0a1441ff04f35c5951becd669d3dc5ce03437e5f1e283e6e7cd0f1d738fb
Oct 30 17:39:38.730978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.730934    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d" event={"ID":"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc","Type":"ContainerStarted","Data":"717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2"}
Oct 30 17:39:38.731453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.731000    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d" event={"ID":"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc","Type":"ContainerStarted","Data":"1fa73c4c9eae0eb13da89ee6b20bfd6cce410ab5eb11160639364c35e9091744"}
Oct 30 17:39:38.731453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.731070    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:38.733799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.733754    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86" event={"ID":"ba41d0af-fed9-4bdf-9cdd-9226e61b1385","Type":"ContainerStarted","Data":"25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9"}
Oct 30 17:39:38.733981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.733820    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86" event={"ID":"ba41d0af-fed9-4bdf-9cdd-9226e61b1385","Type":"ContainerStarted","Data":"e614588531f31912a816ff2e2cd4398ea3e4bb181762d90bc6dc674c03f1ef9f"}
Oct 30 17:39:38.733981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.733852    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:38.735951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.735916    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq" event={"ID":"6d881eb0-5569-439f-859c-a86297985e58","Type":"ContainerStarted","Data":"9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6"}
Oct 30 17:39:38.736107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.735959    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq" event={"ID":"6d881eb0-5569-439f-859c-a86297985e58","Type":"ContainerStarted","Data":"bcec0a1441ff04f35c5951becd669d3dc5ce03437e5f1e283e6e7cd0f1d738fb"}
Oct 30 17:39:38.754990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.754954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:38.776666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.776593    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d" podStartSLOduration=3.776578592 podStartE2EDuration="3.776578592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:38.77535506 +0000 UTC m=+1584.537398874" watchObservedRunningTime="2026-10-30 17:39:38.776578592 +0000 UTC m=+1584.538622412"
Oct 30 17:39:38.821399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.821338    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq" podStartSLOduration=2.501708098 podStartE2EDuration="2.821322332s" totalImagesPullingTime="319.614234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:38.817293608 +0000 UTC m=+1584.579337430" watchObservedRunningTime="2026-10-30 17:39:38.821322332 +0000 UTC m=+1584.583366142"
Oct 30 17:39:38.863495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:38.863429    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86" podStartSLOduration=3.863410546 podStartE2EDuration="3.863410546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:38.861945669 +0000 UTC m=+1584.623989483" watchObservedRunningTime="2026-10-30 17:39:38.863410546 +0000 UTC m=+1584.625454360"
Oct 30 17:39:39.034234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.034195    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"]
Oct 30 17:39:39.039222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.039189    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d047c6-6vxmh"]
Oct 30 17:39:39.052445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.052416    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:39.395294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.395262    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:39.679319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.679237    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:39.745276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.745246    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:39.775708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.775677    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"]
Oct 30 17:39:39.776657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:39.776603    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy" containerID="cri-o://95652db4d6155482d724f056c3288b1573b013911a130a6c0048cf2f1b999a74" gracePeriod=30
Oct 30 17:39:40.373051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.373012    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e396bd00-76e2-4db9-97fc-4d65d04b4218" path="/var/lib/kubelet/pods/e396bd00-76e2-4db9-97fc-4d65d04b4218/volumes"
Oct 30 17:39:40.512885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.512842    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"]
Oct 30 17:39:40.519252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.519156    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.539417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.539339    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-466900-openshift-default-dockercfg-7w5rk\""
Oct 30 17:39:40.576294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.576250    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"]
Oct 30 17:39:40.576890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.576843    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:40.617601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.617553    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.617819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.617633    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.617897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.617871    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.617956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.617946    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmch\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.618022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.618006    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.618079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.618052    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.618153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.618132    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.618225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.618210    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.618296 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.618237    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.696860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.696765    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:40.719775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.719741    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.719969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.719802    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.719969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.719865    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.719969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.719915    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmch\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.719969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.719964    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720011    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720091    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720161    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720181    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720219    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720229    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.720891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720863    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.721038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720899    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.721038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.720934    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.723364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.723337    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.726054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.726031    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.743082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.743049    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.743238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.743063    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmch\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch\") pod \"e2e-api-a-466900-openshift-default-5489c88979-pfnx4\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") " pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.827081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.827040    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:40.838476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.838445    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:40.950581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:40.950502    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:41.175967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.175926    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"]
Oct 30 17:39:41.179637 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:41.179604    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc06e32_12ae_42a5_991a_2959bdd90cf3.slice/crio-3baa1ec0e878d13188f5f25b27750b28aaf377f0ce38a208a63cca3d18180f40 WatchSource:0}: Error finding container 3baa1ec0e878d13188f5f25b27750b28aaf377f0ce38a208a63cca3d18180f40: Status 404 returned error can't find the container with id 3baa1ec0e878d13188f5f25b27750b28aaf377f0ce38a208a63cca3d18180f40
Oct 30 17:39:41.183178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.183144    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:39:41.183281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.183230    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:39:41.183348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.183277    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:39:41.769432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.769398    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" event={"ID":"fbc06e32-12ae-42a5-991a-2959bdd90cf3","Type":"ContainerStarted","Data":"0ce24f70b5c81d11e4c251526a8c98786c181189ddfa11036584bbb0d4601a86"}
Oct 30 17:39:41.769626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.769462    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" event={"ID":"fbc06e32-12ae-42a5-991a-2959bdd90cf3","Type":"ContainerStarted","Data":"3baa1ec0e878d13188f5f25b27750b28aaf377f0ce38a208a63cca3d18180f40"}
Oct 30 17:39:41.813800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.813735    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podStartSLOduration=1.813721106 podStartE2EDuration="1.813721106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:41.812470908 +0000 UTC m=+1587.574514733" watchObservedRunningTime="2026-10-30 17:39:41.813721106 +0000 UTC m=+1587.575764918"
Oct 30 17:39:41.838896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:41.838865    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:42.493176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.493144    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"]
Oct 30 17:39:42.493436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.493409    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s" podUID="e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" containerName="maas-api" containerID="cri-o://5961bd03e088c1cb9ab23dc455773d28a316462efb1804a8cc54b99eb53dc70c" gracePeriod=30
Oct 30 17:39:42.585966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.585926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:42.939886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.923017    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:42.958145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.958113    5534 generic.go:356] "Generic (PLEG): container finished" podID="e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" containerID="5961bd03e088c1cb9ab23dc455773d28a316462efb1804a8cc54b99eb53dc70c" exitCode=0
Oct 30 17:39:42.959535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.959487    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s" event={"ID":"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5","Type":"ContainerDied","Data":"5961bd03e088c1cb9ab23dc455773d28a316462efb1804a8cc54b99eb53dc70c"}
Oct 30 17:39:42.959676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.959569    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:42.961974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.961958    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:39:42.965771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.965753    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:39:42.970297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.970269    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs\") pod \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") "
Oct 30 17:39:42.970529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.970512    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca\") pod \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") "
Oct 30 17:39:42.970687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.970670    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls\") pod \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") "
Oct 30 17:39:42.970752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.970731    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m\" (UniqueName: \"kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m\") pod \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\" (UID: \"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5\") "
Oct 30 17:39:42.971198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.971175    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca" pod "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" (UID: "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:42.971537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.971519    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:42.973847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.973822    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls" pod "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" (UID: "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:42.975092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.975065    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs" pod "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" (UID: "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:42.976142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:42.976112    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m" pod "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" (UID: "e1e7d7f0-cf67-40f9-a190-85fe1a5012b5"). InnerVolumeSpecName "kube-api-access-x924m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:43.150724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.142873    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:43.150724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.142906    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x924m\" (UniqueName: \"kubernetes.io/projected/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-kube-api-access-x924m\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:43.150724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.142923    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:43.150724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.143747    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:43.258913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.258880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:43.379270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.379236    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:43.415416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.415313    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"]
Oct 30 17:39:43.433698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.429150    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df8b20-kw7kz"]
Oct 30 17:39:43.742073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.741993    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:39:43.869242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.869209    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:43.968455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.968423    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"
Oct 30 17:39:43.968950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.968467    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s" event={"ID":"e1e7d7f0-cf67-40f9-a190-85fe1a5012b5","Type":"ContainerDied","Data":"382e1a896ef8689884cba0a7b6ef2ec64229ddc163306992cc96738940c62a1c"}
Oct 30 17:39:43.968950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:43.968520    5534 scope.go:122] "RemoveContainer" containerID="5961bd03e088c1cb9ab23dc455773d28a316462efb1804a8cc54b99eb53dc70c"
Oct 30 17:39:44.046486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.046414    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"]
Oct 30 17:39:44.056473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.056448    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df8b20-865fd465fd-ppc9s"]
Oct 30 17:39:44.372365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.372335    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f70b26-a623-43d9-8d69-1891cafd0a0b" path="/var/lib/kubelet/pods/64f70b26-a623-43d9-8d69-1891cafd0a0b/volumes"
Oct 30 17:39:44.372815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.372800    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" path="/var/lib/kubelet/pods/e1e7d7f0-cf67-40f9-a190-85fe1a5012b5/volumes"
Oct 30 17:39:44.642601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.642485    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:44.761067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.761007    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:44.995002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.994969    5534 generic.go:356] "Generic (PLEG): container finished" podID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerID="95652db4d6155482d724f056c3288b1573b013911a130a6c0048cf2f1b999a74" exitCode=0
Oct 30 17:39:44.996459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:44.996425    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" event={"ID":"8f3885c7-5694-4c68-9deb-8cc6e801d105","Type":"ContainerDied","Data":"95652db4d6155482d724f056c3288b1573b013911a130a6c0048cf2f1b999a74"}
Oct 30 17:39:45.102903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.102871    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:39:45.104186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.104155    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:45.184963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.184857    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.184963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.184931    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185034    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185102    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185156    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185247    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185278    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185408    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.185576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.185487    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw\") pod \"8f3885c7-5694-4c68-9deb-8cc6e801d105\" (UID: \"8f3885c7-5694-4c68-9deb-8cc6e801d105\") "
Oct 30 17:39:45.187451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.187251    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:45.187845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.187630    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:45.188094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.188061    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:45.188233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.188208    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:45.188363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.188346    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:45.203372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.190006    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:45.203372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.200541    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "kube-api-access-tdqzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:45.203372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.202799    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:39:45.203372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.203324    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token" pod "8f3885c7-5694-4c68-9deb-8cc6e801d105" (UID: "8f3885c7-5694-4c68-9deb-8cc6e801d105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:45.264070 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:39:45.264025    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:39:45.288165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288132    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3885c7-5694-4c68-9deb-8cc6e801d105-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288160    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqzw\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-kube-api-access-tdqzw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288172    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288182    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288191    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288200    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288207    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288215    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3885c7-5694-4c68-9deb-8cc6e801d105-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.288352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.288222    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3885c7-5694-4c68-9deb-8cc6e801d105-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:45.766656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.766614    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:39:45.788406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:45.788380    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:46.004706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.004636    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"
Oct 30 17:39:46.004706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.004679    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq" event={"ID":"8f3885c7-5694-4c68-9deb-8cc6e801d105","Type":"ContainerDied","Data":"ec0deaa939d408d2f079ecc663124d87bad69b5df7d8c148cbd0a3a90c46fb59"}
Oct 30 17:39:46.005234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.004750    5534 scope.go:122] "RemoveContainer" containerID="95652db4d6155482d724f056c3288b1573b013911a130a6c0048cf2f1b999a74"
Oct 30 17:39:46.079940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.079904    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"]
Oct 30 17:39:46.085448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.085408    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-zw9qq"]
Oct 30 17:39:46.375206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.375166    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" path="/var/lib/kubelet/pods/8f3885c7-5694-4c68-9deb-8cc6e801d105/volumes"
Oct 30 17:39:46.394044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.394016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:46.489658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.489612    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"]
Oct 30 17:39:46.491554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.491524    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" containerName="maas-api"
Oct 30 17:39:46.491554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.491554    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" containerName="maas-api"
Oct 30 17:39:46.491762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.491711    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy"
Oct 30 17:39:46.491762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.491722    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy"
Oct 30 17:39:46.491932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.491917    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3885c7-5694-4c68-9deb-8cc6e801d105" containerName="istio-proxy"
Oct 30 17:39:46.491970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.491938    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e7d7f0-cf67-40f9-a190-85fe1a5012b5" containerName="maas-api"
Oct 30 17:39:46.497368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.497348    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.512695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.512626    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-466900\""
Oct 30 17:39:46.517304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.517286    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:46.518295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.518277    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"]
Oct 30 17:39:46.570131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.570077    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"]
Oct 30 17:39:46.575972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.575945    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.591718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.591696    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-466900-dockercfg-sbdwc\""
Oct 30 17:39:46.596192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.596157    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"]
Oct 30 17:39:46.624702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.610336    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466900-7c8856645b-l4hdl\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") " pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.624702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.610439    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzt4\" (UniqueName: \"kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4\") pod \"payload-processing-e2e-api-a-466900-7c8856645b-l4hdl\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") " pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.732729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.713438    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzt4\" (UniqueName: \"kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4\") pod \"payload-processing-e2e-api-a-466900-7c8856645b-l4hdl\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") " pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.732729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.713700    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ctq\" (UniqueName: \"kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq\") pod \"payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.732729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.713855    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.732729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.713895    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466900-7c8856645b-l4hdl\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") " pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.732729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.714989    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466900-7c8856645b-l4hdl\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") " pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.836021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.835978    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.836588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.836563    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ctq\" (UniqueName: \"kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq\") pod \"payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.838127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.838104    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.843600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.843574    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzt4\" (UniqueName: \"kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4\") pod \"payload-processing-e2e-api-a-466900-7c8856645b-l4hdl\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") " pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:46.865007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.864974    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ctq\" (UniqueName: \"kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq\") pod \"payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:46.909909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:46.908050    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:47.112842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:47.112816    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:47.230530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:47.230498    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"]
Oct 30 17:39:47.231562 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:47.231538    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae839e0e_9fca_4ca5_8dbf_9ae0b43b22cc.slice/crio-e16261e28c2de32f5ea709a2da0edc396b028069b2334a7a319b182b5a163792 WatchSource:0}: Error finding container e16261e28c2de32f5ea709a2da0edc396b028069b2334a7a319b182b5a163792: Status 404 returned error can't find the container with id e16261e28c2de32f5ea709a2da0edc396b028069b2334a7a319b182b5a163792
Oct 30 17:39:47.426849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:47.426824    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"]
Oct 30 17:39:47.426993 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:47.426971    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e279f3e_ea68_407b_a205_d437cb4f8433.slice/crio-2362da12e642e19d1778efe552d734bd86498ed98255ac8e554239846d3b60c5 WatchSource:0}: Error finding container 2362da12e642e19d1778efe552d734bd86498ed98255ac8e554239846d3b60c5: Status 404 returned error can't find the container with id 2362da12e642e19d1778efe552d734bd86498ed98255ac8e554239846d3b60c5
Oct 30 17:39:47.837005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:47.836972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:48.024963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.024866    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl" event={"ID":"6e279f3e-ea68-407b-a205-d437cb4f8433","Type":"ContainerStarted","Data":"319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec"}
Oct 30 17:39:48.024963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.024925    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl" event={"ID":"6e279f3e-ea68-407b-a205-d437cb4f8433","Type":"ContainerStarted","Data":"2362da12e642e19d1778efe552d734bd86498ed98255ac8e554239846d3b60c5"}
Oct 30 17:39:48.024963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.024952    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:48.026600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.026572    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd" event={"ID":"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc","Type":"ContainerStarted","Data":"605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2"}
Oct 30 17:39:48.026754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.026633    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd" event={"ID":"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc","Type":"ContainerStarted","Data":"e16261e28c2de32f5ea709a2da0edc396b028069b2334a7a319b182b5a163792"}
Oct 30 17:39:48.026754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.026700    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:39:48.066852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.066792    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl" podStartSLOduration=2.066773927 podStartE2EDuration="2.066773927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:48.064369809 +0000 UTC m=+1593.826413623" watchObservedRunningTime="2026-10-30 17:39:48.066773927 +0000 UTC m=+1593.828817740"
Oct 30 17:39:48.105739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:48.105677    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd" podStartSLOduration=2.105658298 podStartE2EDuration="2.105658298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:48.103974792 +0000 UTC m=+1593.866018602" watchObservedRunningTime="2026-10-30 17:39:48.105658298 +0000 UTC m=+1593.867702105"
Oct 30 17:39:49.469732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.469699    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"]
Oct 30 17:39:49.475473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.475443    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.497882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.497857    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"]
Oct 30 17:39:49.500302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.500274    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-81e9336e-serving-cert\""
Oct 30 17:39:49.578695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.578628    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.578866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.578836    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlc4l\" (UniqueName: \"kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.578938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.578869    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.579178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.579156    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.582025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.581994    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:49.692398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.690316    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc4l\" (UniqueName: \"kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.692398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.690376    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.692398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.690574    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.692398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.690628    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.692398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.692080    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.702008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.701498    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:49.727710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.725831    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.727710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.727086    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlc4l\" (UniqueName: \"kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.737492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.730116    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls\") pod \"maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.749206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.749177    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:39:49.750804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.750755    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:39:49.802031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.802001    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:49.821624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:49.821598    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:50.136817 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:50.136784    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002d0148_d056_4e51_bc8d_234b4e50dfac.slice/crio-0676f385f04901ff01eaf73fc428193c9c05457ebf1de179d4dee6222ffbe203 WatchSource:0}: Error finding container 0676f385f04901ff01eaf73fc428193c9c05457ebf1de179d4dee6222ffbe203: Status 404 returned error can't find the container with id 0676f385f04901ff01eaf73fc428193c9c05457ebf1de179d4dee6222ffbe203
Oct 30 17:39:50.137123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:50.137096    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"]
Oct 30 17:39:50.571625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:50.571600    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:50.684710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:50.684672    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:51.007011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.006966    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:51.071145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.071107    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7" event={"ID":"002d0148-d056-4e51-bc8d-234b4e50dfac","Type":"ContainerStarted","Data":"1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4"}
Oct 30 17:39:51.071352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.071175    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7" event={"ID":"002d0148-d056-4e51-bc8d-234b4e50dfac","Type":"ContainerStarted","Data":"0676f385f04901ff01eaf73fc428193c9c05457ebf1de179d4dee6222ffbe203"}
Oct 30 17:39:51.072350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.072327    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:51.123107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.123053    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7" podStartSLOduration=1.7951073549999998 podStartE2EDuration="2.123037296s" totalImagesPullingTime="327.929941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:51.120041614 +0000 UTC m=+1596.882085439" watchObservedRunningTime="2026-10-30 17:39:51.123037296 +0000 UTC m=+1596.885081109"
Oct 30 17:39:51.827506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.827473    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"]
Oct 30 17:39:51.832004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.831987    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:51.848194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.848170    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:39:51.848768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.848747    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:39:51.852585 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.852565    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"]
Oct 30 17:39:51.931733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.931699    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:51.931930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.931753    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gz2n\" (UniqueName: \"kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:51.932094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.932071    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:51.933224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:51.933202    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:52.033228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.033193    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.033397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.033298    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.033397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.033337    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gz2n\" (UniqueName: \"kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.033960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.033934    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.036015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.035994    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.058058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.058029    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gz2n\" (UniqueName: \"kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n\") pod \"maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.147512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.147478    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:52.461083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:52.461055    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"]
Oct 30 17:39:52.462182 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:39:52.462149    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf749ba3_be81_4690_ad61_34e79a383368.slice/crio-78f82f442c1cd4b94bef9cf37e674bb34cd5d674bd55ccc3c510a01870bbcd8c WatchSource:0}: Error finding container 78f82f442c1cd4b94bef9cf37e674bb34cd5d674bd55ccc3c510a01870bbcd8c: Status 404 returned error can't find the container with id 78f82f442c1cd4b94bef9cf37e674bb34cd5d674bd55ccc3c510a01870bbcd8c
Oct 30 17:39:53.094100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:53.094059    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v" event={"ID":"df749ba3-be81-4690-ad61-34e79a383368","Type":"ContainerStarted","Data":"b325e7a41098a6185e9931daf8f24d60feb84c6e779135f9a2b2bb19d6b84268"}
Oct 30 17:39:53.094565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:53.094125    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v" event={"ID":"df749ba3-be81-4690-ad61-34e79a383368","Type":"ContainerStarted","Data":"78f82f442c1cd4b94bef9cf37e674bb34cd5d674bd55ccc3c510a01870bbcd8c"}
Oct 30 17:39:53.135158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:53.135068    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v" podStartSLOduration=2.135054823 podStartE2EDuration="2.135054823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:53.133355434 +0000 UTC m=+1598.895399246" watchObservedRunningTime="2026-10-30 17:39:53.135054823 +0000 UTC m=+1598.897098636"
Oct 30 17:39:53.805449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:53.805360    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:54.107225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:54.107190    5534 generic.go:356] "Generic (PLEG): container finished" podID="df749ba3-be81-4690-ad61-34e79a383368" containerID="b325e7a41098a6185e9931daf8f24d60feb84c6e779135f9a2b2bb19d6b84268" exitCode=0
Oct 30 17:39:54.107626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:54.107275    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v" event={"ID":"df749ba3-be81-4690-ad61-34e79a383368","Type":"ContainerDied","Data":"b325e7a41098a6185e9931daf8f24d60feb84c6e779135f9a2b2bb19d6b84268"}
Oct 30 17:39:55.288040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.288008    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:55.387393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.387294    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca\") pod \"df749ba3-be81-4690-ad61-34e79a383368\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") "
Oct 30 17:39:55.387578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.387431    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca\") pod \"df749ba3-be81-4690-ad61-34e79a383368\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") "
Oct 30 17:39:55.387578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.387487    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n\" (UniqueName: \"kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n\") pod \"df749ba3-be81-4690-ad61-34e79a383368\" (UID: \"df749ba3-be81-4690-ad61-34e79a383368\") "
Oct 30 17:39:55.388046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.388018    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca" pod "df749ba3-be81-4690-ad61-34e79a383368" (UID: "df749ba3-be81-4690-ad61-34e79a383368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:39:55.388604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.388582    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df749ba3-be81-4690-ad61-34e79a383368-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:55.390375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.390348    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n" pod "df749ba3-be81-4690-ad61-34e79a383368" (UID: "df749ba3-be81-4690-ad61-34e79a383368"). InnerVolumeSpecName "kube-api-access-6gz2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:55.390570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.390542    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca" pod "df749ba3-be81-4690-ad61-34e79a383368" (UID: "df749ba3-be81-4690-ad61-34e79a383368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:55.490717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.490683    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df749ba3-be81-4690-ad61-34e79a383368-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:55.490717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.490718    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gz2n\" (UniqueName: \"kubernetes.io/projected/df749ba3-be81-4690-ad61-34e79a383368-kube-api-access-6gz2n\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:39:55.529181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:55.529145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:56.124506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:56.124469    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v" event={"ID":"df749ba3-be81-4690-ad61-34e79a383368","Type":"ContainerDied","Data":"78f82f442c1cd4b94bef9cf37e674bb34cd5d674bd55ccc3c510a01870bbcd8c"}
Oct 30 17:39:56.124506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:56.124510    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"
Oct 30 17:39:56.124790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:56.124527    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f82f442c1cd4b94bef9cf37e674bb34cd5d674bd55ccc3c510a01870bbcd8c"
Oct 30 17:39:57.341173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:57.341139    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:58.100846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:58.100817    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:39:59.011752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:59.011713    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:39:59.045310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:59.045278    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:39:59.052805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:39:59.052753    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:40:00.714791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:00.714753    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:00.913969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:00.913933    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:02.419202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:02.419172    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:02.530896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:02.530858    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:04.224750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:04.223277    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:05.845866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:05.845823    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:05.963230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:05.960904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:07.222713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:07.222682    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:07.506991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:07.506887    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:09.407812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:09.407756    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:11.135893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:11.135846    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:12.992667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:12.992080    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:14.928067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:14.928032    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:15.247279 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:40:15.247192    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:40:15.454601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:15.454570    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:16.155219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.155176    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:16.460995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.460913    5534 scope.go:122] "RemoveContainer" containerID="01cd637a0b60bd0829b5f1dce4f5c5638e360e292ee5e40e76bccd41a6e62d64"
Oct 30 17:40:16.468073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.468035    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"]
Oct 30 17:40:16.468804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.468788    5534 cpu_manager.go:414] "containerMap: removing container" podUID="df749ba3-be81-4690-ad61-34e79a383368" containerName="revoke-keys"
Oct 30 17:40:16.468865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.468805    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df749ba3-be81-4690-ad61-34e79a383368" containerName="revoke-keys"
Oct 30 17:40:16.468930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.468921    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="df749ba3-be81-4690-ad61-34e79a383368" containerName="revoke-keys"
Oct 30 17:40:16.474120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.474091    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.474736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.474714    5534 scope.go:122] "RemoveContainer" containerID="066136b9d46fb306872d0d958a23c40ddaec73567cf4500896263c4e9a71fed7"
Oct 30 17:40:16.486601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.486578    5534 scope.go:122] "RemoveContainer" containerID="f37c0228dbb26b0cb273bef00995341e8335f7f8fe4694eb3ea968d81f0429f4"
Oct 30 17:40:16.489517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.489472    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-466900-serving-cert\""
Oct 30 17:40:16.499030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.498997    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"]
Oct 30 17:40:16.502189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.502166    5534 scope.go:122] "RemoveContainer" containerID="7ac41ead9489155d69b56cedad040f64a1d28db9bf6beff60113108271e8367e"
Oct 30 17:40:16.517892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.517863    5534 scope.go:122] "RemoveContainer" containerID="5681e9aee60d03b64eab5caa9363dcc2b8be0cee9742ab8f1e0891606a468228"
Oct 30 17:40:16.531310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.531275    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:16.534836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.534809    5534 scope.go:122] "RemoveContainer" containerID="caee496daea8d78d94d77fd507cbbae48cdfc80c65e646d0e9b8e95671a8b448"
Oct 30 17:40:16.559519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.559478    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.559759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.559548    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.559759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.559577    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.559759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.559737    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5kl\" (UniqueName: \"kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.560158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.560139    5534 scope.go:122] "RemoveContainer" containerID="e382d8cf5d50f05ed9d84c67bad06322e072b060e20aabbea23c7f6fcc5be67e"
Oct 30 17:40:16.655893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.655349    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.661012    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5kl\" (UniqueName: \"kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.661187    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.661229    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.661247    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.661973    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:40:16.662411    5534 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-466900-serving-cert: secret "maas-api-e2e-api-a-466900-serving-cert" not found
Oct 30 17:40:16.666388 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:40:16.662489    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls podName:d74d0783-6885-41b0-89d8-f5989412a3e3 nodeName:}" failed. No retries permitted until 2026-10-30 17:40:17.162466833 +0000 UTC m=+1622.924510645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls") pod "maas-api-e2e-api-a-466900-c99899c59-swpc9" (UID: "d74d0783-6885-41b0-89d8-f5989412a3e3") : secret "maas-api-e2e-api-a-466900-serving-cert" not found
Oct 30 17:40:16.668358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.668323    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.688350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.688317    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5kl\" (UniqueName: \"kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:16.793781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.793676    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:16.986511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:16.986463    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:17.170152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.170114    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:17.173060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.173027    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\") pod \"maas-api-e2e-api-a-466900-c99899c59-swpc9\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:17.240669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.238762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:17.392710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.392628    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:17.651875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.651825    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"]
Oct 30 17:40:17.661086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.661058    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.694713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.694678    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"]
Oct 30 17:40:17.697226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.697194    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.697519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.697492    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.697696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.697679    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.697839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.697824    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.697984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.697964    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62jl4\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.698372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.698346    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.698719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.698557    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.698719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.698607    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.699204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.699046    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.701370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.701312    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cc4e96-openshift-default-dockercfg-lc6vn\""
Oct 30 17:40:17.726019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.725926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:17.738593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.738550    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"]
Oct 30 17:40:17.748904 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:40:17.748869    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74d0783_6885_41b0_89d8_f5989412a3e3.slice/crio-61ec0c10ef9bc0ff0d826ce31546a842dd9675ae974b426a1ad3c19bdf04c3d4 WatchSource:0}: Error finding container 61ec0c10ef9bc0ff0d826ce31546a842dd9675ae974b426a1ad3c19bdf04c3d4: Status 404 returned error can't find the container with id 61ec0c10ef9bc0ff0d826ce31546a842dd9675ae974b426a1ad3c19bdf04c3d4
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.803445    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.804047    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.804390    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.804453    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805020    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805065    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805060    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805155    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805221    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62jl4\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805282    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.805670    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.806001    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.806002    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.806740    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.839706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.810987    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.866562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.853001    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62jl4\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.892666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.874297    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:17.892666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.887577    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:17.904062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:17.903967    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy\") pod \"e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") " pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:18.015725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.010431    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:18.343434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.343388    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9" event={"ID":"d74d0783-6885-41b0-89d8-f5989412a3e3","Type":"ContainerStarted","Data":"bed0e4ae2a3231a2b551ce16df45e3907705ce1ee2a684f93c53d1d49a6092b1"}
Oct 30 17:40:18.344037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.343458    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9" event={"ID":"d74d0783-6885-41b0-89d8-f5989412a3e3","Type":"ContainerStarted","Data":"61ec0c10ef9bc0ff0d826ce31546a842dd9675ae974b426a1ad3c19bdf04c3d4"}
Oct 30 17:40:18.344037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.343754    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:18.375453 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:40:18.375404    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1f1ddd_0676_4413_9dd2_1ad060ec5090.slice/crio-ecb060e51180abb6db736587d7675cde11181aa23e6a8ec365840f51aa05f0f1 WatchSource:0}: Error finding container ecb060e51180abb6db736587d7675cde11181aa23e6a8ec365840f51aa05f0f1: Status 404 returned error can't find the container with id ecb060e51180abb6db736587d7675cde11181aa23e6a8ec365840f51aa05f0f1
Oct 30 17:40:18.376249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.376220    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"]
Oct 30 17:40:18.379015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.378975    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:40:18.379107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.379067    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:40:18.379150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.379111    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:40:18.390255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.390201    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9" podStartSLOduration=2.002357629 podStartE2EDuration="2.390181562s" totalImagesPullingTime="387.823933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:18.38715843 +0000 UTC m=+1624.149202244" watchObservedRunningTime="2026-10-30 17:40:18.390181562 +0000 UTC m=+1624.152225373"
Oct 30 17:40:18.749312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:18.749230    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:19.354106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:19.354070    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" event={"ID":"af1f1ddd-0676-4413-9dd2-1ad060ec5090","Type":"ContainerStarted","Data":"ff695258dd354590231a26c6c6e6a8130ed7971cd2d60abb5639d2e785887f4d"}
Oct 30 17:40:19.354530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:19.354135    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" event={"ID":"af1f1ddd-0676-4413-9dd2-1ad060ec5090","Type":"ContainerStarted","Data":"ecb060e51180abb6db736587d7675cde11181aa23e6a8ec365840f51aa05f0f1"}
Oct 30 17:40:19.415214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:19.415162    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podStartSLOduration=2.415147571 podStartE2EDuration="2.415147571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:19.412288696 +0000 UTC m=+1625.174332508" watchObservedRunningTime="2026-10-30 17:40:19.415147571 +0000 UTC m=+1625.177191430"
Oct 30 17:40:20.012080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:20.012038    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:20.019181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:20.019154    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:20.378278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:20.378243    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:20.378847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:20.378342    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:40:20.630227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:20.630139    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:20.749396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:20.749359    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:22.213584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.213543    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:22.342192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.342130    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:22.454985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.454944    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"]
Oct 30 17:40:22.460813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.460778    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:22.462271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.462250    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:22.480142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.480117    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"]
Oct 30 17:40:22.487785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.487762    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-49e155-dockercfg-mjkc4\""
Oct 30 17:40:22.562718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.555020    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-49e155\""
Oct 30 17:40:22.711704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.704551    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplbl\" (UniqueName: \"kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl\") pod \"payload-processing-e2e-api-b-49e155-8594b46674-dlx7k\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") " pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:22.711704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.704628    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-49e155-8594b46674-dlx7k\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") " pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:22.810564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.809172    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplbl\" (UniqueName: \"kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl\") pod \"payload-processing-e2e-api-b-49e155-8594b46674-dlx7k\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") " pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:22.810564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.809233    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-49e155-8594b46674-dlx7k\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") " pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:22.810564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.810459    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-49e155-8594b46674-dlx7k\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") " pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:22.819094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.819037    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:22.836851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:22.836818    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplbl\" (UniqueName: \"kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl\") pod \"payload-processing-e2e-api-b-49e155-8594b46674-dlx7k\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") " pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:23.083412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:23.083320    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:23.402221 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:40:23.402189    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3967f1_65d5_472f_a4ec_e05da2fdd9d1.slice/crio-c8784ea7c904c67708d364a7919737575d6aceece437e70b4c32ced8daaa4ccd WatchSource:0}: Error finding container c8784ea7c904c67708d364a7919737575d6aceece437e70b4c32ced8daaa4ccd: Status 404 returned error can't find the container with id c8784ea7c904c67708d364a7919737575d6aceece437e70b4c32ced8daaa4ccd
Oct 30 17:40:23.402612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:23.402560    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"]
Oct 30 17:40:24.215630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.215588    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:24.340166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.340124    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:24.396063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.395824    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:40:24.396063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.395878    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"]
Oct 30 17:40:24.412350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.412309    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.415591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.415553    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"]
Oct 30 17:40:24.430028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.429979    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc4e96\""
Oct 30 17:40:24.430217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.430193    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cc4e96-dockercfg-zvggv\""
Oct 30 17:40:24.432361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.431437    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k" event={"ID":"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1","Type":"ContainerStarted","Data":"ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a"}
Oct 30 17:40:24.432361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.431501    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k" event={"ID":"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1","Type":"ContainerStarted","Data":"c8784ea7c904c67708d364a7919737575d6aceece437e70b4c32ced8daaa4ccd"}
Oct 30 17:40:24.432565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.432435    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:24.434398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.434369    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxhj\" (UniqueName: \"kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj\") pod \"payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.434699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.434575    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.536356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.536253    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxhj\" (UniqueName: \"kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj\") pod \"payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.537216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.537186    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.537756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.537179    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k" podStartSLOduration=2.537161497 podStartE2EDuration="2.537161497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:24.53508751 +0000 UTC m=+1630.297131324" watchObservedRunningTime="2026-10-30 17:40:24.537161497 +0000 UTC m=+1630.299205311"
Oct 30 17:40:24.538418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.538387    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.564474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.564434    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxhj\" (UniqueName: \"kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj\") pod \"payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.760071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.760025    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:24.909297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:24.908960    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:25.159746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:25.159713    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"]
Oct 30 17:40:25.441170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:25.441072    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s" event={"ID":"8a3a7310-8efd-426c-beed-da67d7bb5354","Type":"ContainerStarted","Data":"a2ff37051ec1aaa9722696b6901df0073bb988552ba98adb0c30a6fecac8f216"}
Oct 30 17:40:25.441170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:25.441139    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s" event={"ID":"8a3a7310-8efd-426c-beed-da67d7bb5354","Type":"ContainerStarted","Data":"bd87f25f9da219410ed48ad33355293835de7bba51f03f0afd85af10cd5e6439"}
Oct 30 17:40:25.441170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:25.441168    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:25.484022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:25.483968    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s" podStartSLOduration=1.483952491 podStartE2EDuration="1.483952491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:25.483016914 +0000 UTC m=+1631.245060728" watchObservedRunningTime="2026-10-30 17:40:25.483952491 +0000 UTC m=+1631.245996304"
Oct 30 17:40:26.714667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:26.714531    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:28.509056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:28.509024    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:30.273005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:30.272972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:31.976871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:31.976833    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:33.589155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:33.589111    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:35.042687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:35.042655    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"]
Oct 30 17:40:35.054046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:35.054006    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c10c48-lncgn"]
Oct 30 17:40:35.353973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:35.353940    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:35.579676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:35.579625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:36.392420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:36.392377    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19758f36-3669-41d9-ac48-5cac16aabfb0" path="/var/lib/kubelet/pods/19758f36-3669-41d9-ac48-5cac16aabfb0/volumes"
Oct 30 17:40:36.533679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:36.532190    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:40:36.536993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:36.536966    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:40:37.610672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:37.608607    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:39.625600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:39.625572    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:41.885708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:41.885671    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:42.081864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:42.081832    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:43.608588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:43.608559    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:44.000896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:44.000806    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:40:44.001158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:44.001127    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:40:45.249745 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:40:45.249700    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:40:45.261923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:45.261887    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:45.379190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:45.379155    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:46.031380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.031339    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"]
Oct 30 17:40:46.042787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.042761    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc985012-rxd8k"]
Oct 30 17:40:46.375827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.375793    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e817252-4924-4ab5-8bda-5c39661a1105" path="/var/lib/kubelet/pods/9e817252-4924-4ab5-8bda-5c39661a1105/volumes"
Oct 30 17:40:46.768048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.768011    5534 generic.go:356] "Generic (PLEG): container finished" podID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerID="0ce24f70b5c81d11e4c251526a8c98786c181189ddfa11036584bbb0d4601a86" exitCode=0
Oct 30 17:40:46.768300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.768220    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" event={"ID":"fbc06e32-12ae-42a5-991a-2959bdd90cf3","Type":"ContainerDied","Data":"0ce24f70b5c81d11e4c251526a8c98786c181189ddfa11036584bbb0d4601a86"}
Oct 30 17:40:46.768790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.768768    5534 scope.go:122] "RemoveContainer" containerID="0ce24f70b5c81d11e4c251526a8c98786c181189ddfa11036584bbb0d4601a86"
Oct 30 17:40:46.770967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.770937    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:40:46.771086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.771033    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:40:46.771168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:46.771083    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:40:47.192356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:47.192269    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:47.794749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:47.794702    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" event={"ID":"fbc06e32-12ae-42a5-991a-2959bdd90cf3","Type":"ContainerStarted","Data":"2ca5132d0c6db7796a12340bb5975ad003e7ef3b570d0a55bd4bf2261fb2ce41"}
Oct 30 17:40:47.839484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:47.839454    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:40:47.839738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:47.839719    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 17:40:47.839807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:47.839764    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 17:40:48.839702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:48.839666    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 17:40:48.840117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:48.839740    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 17:40:48.941771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:48.941734    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:49.839059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:49.839022    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 17:40:49.839264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:49.839078    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 17:40:50.782809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:50.782775    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:50.839842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:50.839807    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 17:40:50.840017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:50.839866    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 17:40:51.088847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:51.088750    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:51.839783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:51.839749    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 17:40:51.840318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:51.839816    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 17:40:52.839139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:52.839100    5534 patch_prober.go:29] interesting pod/e2e-api-a-466900-openshift-default-5489c88979-pfnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 17:40:52.840363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:52.840333    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 17:40:53.156329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:53.156269    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:53.844026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:53.843996    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:40:53.844477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:53.844461    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:40:53.845114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:53.845092    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:40:54.969395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:54.969356    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:55.077858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:55.077821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:57.261575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:57.261535    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:57.374993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:57.374959    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:57.488462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:57.488431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:40:59.038531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:40:59.038472    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:00.161061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:00.161024    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"]
Oct 30 17:41:00.188078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:00.188044    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"]
Oct 30 17:41:00.291318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:00.291282    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:00.414925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:00.414821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:01.013919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:01.013870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:02.354836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.354799    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"]
Oct 30 17:41:02.355356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.355047    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7" podUID="002d0148-d056-4e51-bc8d-234b4e50dfac" containerName="maas-api" containerID="cri-o://1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4" gracePeriod=30
Oct 30 17:41:02.432054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.432020    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:02.661058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.661021    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:41:02.763678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.760159    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca\") pod \"002d0148-d056-4e51-bc8d-234b4e50dfac\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") "
Oct 30 17:41:02.763678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.760313    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs\") pod \"002d0148-d056-4e51-bc8d-234b4e50dfac\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") "
Oct 30 17:41:02.763678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.760413    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls\") pod \"002d0148-d056-4e51-bc8d-234b4e50dfac\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") "
Oct 30 17:41:02.763678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.760451    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l\" (UniqueName: \"kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l\") pod \"002d0148-d056-4e51-bc8d-234b4e50dfac\" (UID: \"002d0148-d056-4e51-bc8d-234b4e50dfac\") "
Oct 30 17:41:02.763678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.760619    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca" pod "002d0148-d056-4e51-bc8d-234b4e50dfac" (UID: "002d0148-d056-4e51-bc8d-234b4e50dfac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:02.763678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.761227    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002d0148-d056-4e51-bc8d-234b4e50dfac-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:02.764890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.764857    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs" pod "002d0148-d056-4e51-bc8d-234b4e50dfac" (UID: "002d0148-d056-4e51-bc8d-234b4e50dfac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:02.765046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.764952    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls" pod "002d0148-d056-4e51-bc8d-234b4e50dfac" (UID: "002d0148-d056-4e51-bc8d-234b4e50dfac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:02.765046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.765023    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l" pod "002d0148-d056-4e51-bc8d-234b4e50dfac" (UID: "002d0148-d056-4e51-bc8d-234b4e50dfac"). InnerVolumeSpecName "kube-api-access-dlc4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:02.876736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.875477    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:02.876736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.875512    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002d0148-d056-4e51-bc8d-234b4e50dfac-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:02.876736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.875526    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlc4l\" (UniqueName: \"kubernetes.io/projected/002d0148-d056-4e51-bc8d-234b4e50dfac-kube-api-access-dlc4l\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:02.946666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:02.946547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:03.046678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.032896    5534 generic.go:356] "Generic (PLEG): container finished" podID="002d0148-d056-4e51-bc8d-234b4e50dfac" containerID="1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4" exitCode=0
Oct 30 17:41:03.046678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.033245    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7" event={"ID":"002d0148-d056-4e51-bc8d-234b4e50dfac","Type":"ContainerDied","Data":"1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4"}
Oct 30 17:41:03.046678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.033307    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7" event={"ID":"002d0148-d056-4e51-bc8d-234b4e50dfac","Type":"ContainerDied","Data":"0676f385f04901ff01eaf73fc428193c9c05457ebf1de179d4dee6222ffbe203"}
Oct 30 17:41:03.046678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.033341    5534 scope.go:122] "RemoveContainer" containerID="1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4"
Oct 30 17:41:03.046678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.033528    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"
Oct 30 17:41:03.069420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.069367    5534 scope.go:122] "RemoveContainer" containerID="1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4"
Oct 30 17:41:03.069797 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:41:03.069775    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4\": container with ID starting with 1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4 not found: ID does not exist" containerID="1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4"
Oct 30 17:41:03.069944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.069808    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4"} err="failed to get container status \"1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4\": rpc error: code = NotFound desc = could not find container \"1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4\": container with ID starting with 1991be5def80553ea4ec74e4bae405f5ae63f5b5e48247f3fb5eb8f1c4cf30f4 not found: ID does not exist"
Oct 30 17:41:03.102864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.102828    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:03.104549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.104514    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"]
Oct 30 17:41:03.113224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.113192    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81e9336e-6d6b9984c-mf5n7"]
Oct 30 17:41:03.212533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.212441    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:03.635224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:03.635190    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:04.374429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:04.374390    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002d0148-d056-4e51-bc8d-234b4e50dfac" path="/var/lib/kubelet/pods/002d0148-d056-4e51-bc8d-234b4e50dfac/volumes"
Oct 30 17:41:04.901107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:04.901078    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:05.162126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.162029    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86" podUID="ba41d0af-fed9-4bdf-9cdd-9226e61b1385" containerName="payload-processing" containerID="cri-o://25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9" gracePeriod=25
Oct 30 17:41:05.189725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.189693    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d" podUID="c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" containerName="payload-pre-processing" containerID="cri-o://717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2" gracePeriod=25
Oct 30 17:41:05.510851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.507134    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:41:05.534403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.534361    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:41:05.644713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.644682    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz\" (UniqueName: \"kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz\") pod \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") "
Oct 30 17:41:05.644911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.644749    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume\") pod \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\" (UID: \"ba41d0af-fed9-4bdf-9cdd-9226e61b1385\") "
Oct 30 17:41:05.644993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.644953    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm\" (UniqueName: \"kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm\") pod \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") "
Oct 30 17:41:05.645068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.645059    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume\") pod \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\" (UID: \"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc\") "
Oct 30 17:41:05.646179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.646147    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume" pod "c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" (UID: "c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:05.646837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.646815    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume" pod "ba41d0af-fed9-4bdf-9cdd-9226e61b1385" (UID: "ba41d0af-fed9-4bdf-9cdd-9226e61b1385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:05.648991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.648958    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm" pod "c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" (UID: "c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc"). InnerVolumeSpecName "kube-api-access-plglm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:05.649834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.649806    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz" pod "ba41d0af-fed9-4bdf-9cdd-9226e61b1385" (UID: "ba41d0af-fed9-4bdf-9cdd-9226e61b1385"). InnerVolumeSpecName "kube-api-access-6wxmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:05.747816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.747701    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plglm\" (UniqueName: \"kubernetes.io/projected/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-kube-api-access-plglm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:05.747816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.747740    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:05.747816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.747754    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxmz\" (UniqueName: \"kubernetes.io/projected/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-kube-api-access-6wxmz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:05.747816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.747767    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba41d0af-fed9-4bdf-9cdd-9226e61b1385-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:05.921395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:05.921352    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:06.039071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.038951    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:06.063358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.063322    5534 generic.go:356] "Generic (PLEG): container finished" podID="c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" containerID="717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2" exitCode=0
Oct 30 17:41:06.063539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.063395    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"
Oct 30 17:41:06.063539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.063412    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d" event={"ID":"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc","Type":"ContainerDied","Data":"717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2"}
Oct 30 17:41:06.063539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.063517    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d" event={"ID":"c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc","Type":"ContainerDied","Data":"1fa73c4c9eae0eb13da89ee6b20bfd6cce410ab5eb11160639364c35e9091744"}
Oct 30 17:41:06.063815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.063551    5534 scope.go:122] "RemoveContainer" containerID="717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2"
Oct 30 17:41:06.065196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.065167    5534 generic.go:356] "Generic (PLEG): container finished" podID="ba41d0af-fed9-4bdf-9cdd-9226e61b1385" containerID="25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9" exitCode=0
Oct 30 17:41:06.065307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.065236    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"
Oct 30 17:41:06.065436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.065416    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86" event={"ID":"ba41d0af-fed9-4bdf-9cdd-9226e61b1385","Type":"ContainerDied","Data":"25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9"}
Oct 30 17:41:06.065522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.065467    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86" event={"ID":"ba41d0af-fed9-4bdf-9cdd-9226e61b1385","Type":"ContainerDied","Data":"e614588531f31912a816ff2e2cd4398ea3e4bb181762d90bc6dc674c03f1ef9f"}
Oct 30 17:41:06.080099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.080077    5534 scope.go:122] "RemoveContainer" containerID="717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2"
Oct 30 17:41:06.080424 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:41:06.080401    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2\": container with ID starting with 717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2 not found: ID does not exist" containerID="717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2"
Oct 30 17:41:06.080503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.080436    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2"} err="failed to get container status \"717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2\": rpc error: code = NotFound desc = could not find container \"717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2\": container with ID starting with 717418658dfa2ab83f1f2eb8b5f517f9dbbd4c8fda69ebf7ae4067d1507c5bb2 not found: ID does not exist"
Oct 30 17:41:06.080503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.080458    5534 scope.go:122] "RemoveContainer" containerID="25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9"
Oct 30 17:41:06.091607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.091585    5534 scope.go:122] "RemoveContainer" containerID="25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9"
Oct 30 17:41:06.091930 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:41:06.091907    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9\": container with ID starting with 25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9 not found: ID does not exist" containerID="25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9"
Oct 30 17:41:06.091993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.091943    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9"} err="failed to get container status \"25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9\": rpc error: code = NotFound desc = could not find container \"25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9\": container with ID starting with 25b3a68073561305effffb6d89846af1625963d5440e7975fb340055a16197a9 not found: ID does not exist"
Oct 30 17:41:06.120316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.120286    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"]
Oct 30 17:41:06.123652 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.123619    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-t2d86"]
Oct 30 17:41:06.156042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.156012    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"]
Oct 30 17:41:06.162063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.162019    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81e9336e-854bcdf7dc-qzt2d"]
Oct 30 17:41:06.375011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.374973    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba41d0af-fed9-4bdf-9cdd-9226e61b1385" path="/var/lib/kubelet/pods/ba41d0af-fed9-4bdf-9cdd-9226e61b1385/volumes"
Oct 30 17:41:06.375688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.375665    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" path="/var/lib/kubelet/pods/c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc/volumes"
Oct 30 17:41:06.600946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.600904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:06.710808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:06.710726    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:07.044074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044005    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"]
Oct 30 17:41:07.044837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044819    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ba41d0af-fed9-4bdf-9cdd-9226e61b1385" containerName="payload-processing"
Oct 30 17:41:07.044891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044839    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba41d0af-fed9-4bdf-9cdd-9226e61b1385" containerName="payload-processing"
Oct 30 17:41:07.044891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044862    5534 cpu_manager.go:414] "containerMap: removing container" podUID="002d0148-d056-4e51-bc8d-234b4e50dfac" containerName="maas-api"
Oct 30 17:41:07.044891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044867    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002d0148-d056-4e51-bc8d-234b4e50dfac" containerName="maas-api"
Oct 30 17:41:07.044996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044894    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" containerName="payload-pre-processing"
Oct 30 17:41:07.044996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044900    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" containerName="payload-pre-processing"
Oct 30 17:41:07.045058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.044999    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ed58ec-1cbd-472b-b0a8-960cf40aa5cc" containerName="payload-pre-processing"
Oct 30 17:41:07.045058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.045012    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba41d0af-fed9-4bdf-9cdd-9226e61b1385" containerName="payload-processing"
Oct 30 17:41:07.045058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.045021    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="002d0148-d056-4e51-bc8d-234b4e50dfac" containerName="maas-api"
Oct 30 17:41:07.050047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.050023    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.086924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.086889    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"]
Oct 30 17:41:07.121321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.121289    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-74762d-openshift-default-dockercfg-92brx\""
Oct 30 17:41:07.122633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.122592    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:07.181533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.181485    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.181762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.181547    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.181838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.181815    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.181915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.181878    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.181915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.181910    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.182122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.181960    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.182122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.182071    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.182122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.182113    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2d8h\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.182283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.182175    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.237889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.237857    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:07.296710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.295160    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309042    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309244    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309305    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309330    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309396    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309464    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309498    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2d8h\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.309573    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.310197    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.310983    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.311231    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.311447    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.312083    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.317175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.315679    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.322662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.319206    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.364068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.363997    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2d8h\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.384677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.371198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.391701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.389954    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:07.779548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.779517    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"]
Oct 30 17:41:07.780978 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:07.780946    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c0f38e_2d80_4cfb_9bc5_70dcaf27011b.slice/crio-3da1389ec3e3f5c4306959a8f3d55812536ef4b2b1c750902979a9b89e82c18a WatchSource:0}: Error finding container 3da1389ec3e3f5c4306959a8f3d55812536ef4b2b1c750902979a9b89e82c18a: Status 404 returned error can't find the container with id 3da1389ec3e3f5c4306959a8f3d55812536ef4b2b1c750902979a9b89e82c18a
Oct 30 17:41:07.784186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.784150    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:07.784283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.784234    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:07.784283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:07.784266    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:08.096213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.096172    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" event={"ID":"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b","Type":"ContainerStarted","Data":"74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"}
Oct 30 17:41:08.096619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.096239    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" event={"ID":"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b","Type":"ContainerStarted","Data":"3da1389ec3e3f5c4306959a8f3d55812536ef4b2b1c750902979a9b89e82c18a"}
Oct 30 17:41:08.143098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.143050    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podStartSLOduration=1.143035259 podStartE2EDuration="1.143035259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:08.140157518 +0000 UTC m=+1673.902201376" watchObservedRunningTime="2026-10-30 17:41:08.143035259 +0000 UTC m=+1673.905079071"
Oct 30 17:41:08.391085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.391001    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:08.397832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.397804    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:08.658136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.658047    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:08.771992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:08.771954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:09.104952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.104920    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:09.105981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.105963    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:41:09.287758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.287724    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"]
Oct 30 17:41:09.295279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.295252    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"]
Oct 30 17:41:09.295447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.295395    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.301235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.301212    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.330930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.327463    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"]
Oct 30 17:41:09.335320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.335291    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"]
Oct 30 17:41:09.350463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350426    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.350697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350485    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.350697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350509    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpphp\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.350834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350708    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.350834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350760    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.350834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350798    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.350834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350828    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350870    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350911    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.350992    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351039    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351083    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351152    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351260    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351318    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351368    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ssp\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351443    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.351554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.351531    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453109    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453179    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453217    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453241    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpphp\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453274    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453312    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453346    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453400    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453446    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453486    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453557    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453556    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453624    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453720    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453790    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.453897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453898    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.454287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453951    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.454287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.453998    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ssp\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.454287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454065    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.454287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454158    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.454510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454227    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.454510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454417    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.454510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454462    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.454729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454602    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.454729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454656    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.454961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.454873    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.455265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.455245    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.455370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.455346    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.456907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.456702    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.457906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.457621    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.457906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.457820    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.458076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.457960    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.476882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.476834    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpphp\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.477041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.476896    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.477221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.477202    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.477544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.477518    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ssp\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-9nddk\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.629043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.629006    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:09.640833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.640804    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:09.982278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.982253    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"]
Oct 30 17:41:09.984740 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:09.984707    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ff310f_c0ac_4b45_b91c_1433f37caa27.slice/crio-b09415531cc700c8e95cf0ea367a54dbf4fa8de719c43250a4e87b00c123ad44 WatchSource:0}: Error finding container b09415531cc700c8e95cf0ea367a54dbf4fa8de719c43250a4e87b00c123ad44: Status 404 returned error can't find the container with id b09415531cc700c8e95cf0ea367a54dbf4fa8de719c43250a4e87b00c123ad44
Oct 30 17:41:09.988691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.988629    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:09.988797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.988739    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:09.988797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.988783    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:09.995001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:09.993209    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"]
Oct 30 17:41:10.001480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.001445    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:10.001575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.001526    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:10.001575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.001570    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:10.124333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.124300    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" event={"ID":"b31f0143-e325-44c1-9803-af5c1708e079","Type":"ContainerStarted","Data":"3413cbe5e0a3095396d6c188dc2b9ee0234f1134fa418a7c3090bbd955b5906c"}
Oct 30 17:41:10.124798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.124354    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" event={"ID":"b31f0143-e325-44c1-9803-af5c1708e079","Type":"ContainerStarted","Data":"967f841580f1014326d25d8df3de5e376a734af76dd3e43bc69b1000d0bbdd90"}
Oct 30 17:41:10.126904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.126875    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" event={"ID":"f5ff310f-c0ac-4b45-b91c-1433f37caa27","Type":"ContainerStarted","Data":"8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c"}
Oct 30 17:41:10.127062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.126924    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" event={"ID":"f5ff310f-c0ac-4b45-b91c-1433f37caa27","Type":"ContainerStarted","Data":"b09415531cc700c8e95cf0ea367a54dbf4fa8de719c43250a4e87b00c123ad44"}
Oct 30 17:41:10.180674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.180385    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podStartSLOduration=1.180363935 podStartE2EDuration="1.180363935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:10.171717939 +0000 UTC m=+1675.933761753" watchObservedRunningTime="2026-10-30 17:41:10.180363935 +0000 UTC m=+1675.942407749"
Oct 30 17:41:10.221793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.221734    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podStartSLOduration=1.22171793 podStartE2EDuration="1.22171793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:10.219047331 +0000 UTC m=+1675.981091146" watchObservedRunningTime="2026-10-30 17:41:10.22171793 +0000 UTC m=+1675.983761743"
Oct 30 17:41:10.358403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.358364    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:10.629677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.629592    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:10.640991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:10.640971    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:11.068351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:11.068245    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:11.185673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:11.185619    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:11.631300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:11.631266    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 17:41:11.631586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:11.631561    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 17:41:11.645860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:11.645826    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 17:41:11.646062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:11.646044    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 17:41:12.247315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:12.247283    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:12.629357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:12.629322    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 17:41:12.629529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:12.629383    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 17:41:12.641374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:12.641342    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 17:41:12.641552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:12.641417    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 17:41:13.629843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:13.629806    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 17:41:13.630268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:13.629882    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 17:41:13.642029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:13.641998    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 17:41:13.642169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:13.642068    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 17:41:14.441686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:14.441635    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:14.640083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:14.640043    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 17:41:14.642315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:14.642285    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 17:41:14.642499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:14.641259    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 17:41:14.642679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:14.642655    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 17:41:15.249894 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:41:15.249794    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:41:15.630028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:15.629989    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 17:41:15.630273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:15.630251    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 17:41:15.641922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:15.641893    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 17:41:15.642438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:15.641941    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 17:41:16.349767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.349735    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:16.630161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.630066    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 17:41:16.630161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.630131    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 17:41:16.641142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.641112    5534 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 17:41:16.641274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.641155    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 17:41:16.706845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.706801    5534 scope.go:122] "RemoveContainer" containerID="159b55a66c4ff45489c2a8fcdd27ebe5bd709929958bc55fbc23aa531a42e566"
Oct 30 17:41:16.718382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.718359    5534 scope.go:122] "RemoveContainer" containerID="559bc3c4da267dd1df4bbd1938aa657e0589d1303462439bde8b5a21006abe20"
Oct 30 17:41:16.729561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:16.729533    5534 scope.go:122] "RemoveContainer" containerID="7792d8d63477b1ebb399e1f9114daaebfa6cfd27cbffdceef5908913dd4cab5f"
Oct 30 17:41:17.635733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:17.635687    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:17.636269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:17.636243    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:17.637218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:17.637199    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:17.647002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:17.646975    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:17.647719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:17.647697    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:17.649281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:17.649252    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:18.080724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:18.080427    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:18.611525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:18.609986    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:18.749665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:18.744131    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:18.769693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:18.767235    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"]
Oct 30 17:41:18.775252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:18.775208    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1c973-f6t5v"]
Oct 30 17:41:18.859577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:18.859545    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:19.288571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:19.288523    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:20.374078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:20.374046    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df749ba3-be81-4690-ad61-34e79a383368" path="/var/lib/kubelet/pods/df749ba3-be81-4690-ad61-34e79a383368/volumes"
Oct 30 17:41:21.133335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:21.133301    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:23.074668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:23.073771    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:41:23.074668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:23.073860    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:41:23.251466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:23.251426    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:25.106136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:25.106093    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:25.735397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:25.735363    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:25.848927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:25.848896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:26.137025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:26.136980    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:26.980068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:26.980012    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:27.097772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.097702    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:27.442012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.441980    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"]
Oct 30 17:41:27.449906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.449870    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.480303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.479096    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"]
Oct 30 17:41:27.483988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.483961    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bba90b-openshift-default-dockercfg-d9vst\""
Oct 30 17:41:27.539211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.539175    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:27.599951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.599892    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600016    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600047    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600137    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600359    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600445    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600493    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8ms\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.600608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600546    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.601652 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.600613    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.662319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.662282    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717219    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717293    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717329    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717434    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717777    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717834    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717868    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8ms\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717925    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.717954    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.718493    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.719187    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.719788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.719444    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.744884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.740720    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.744884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.741081    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.798809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.781357    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.853760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.850366    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8ms\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.853760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.851939    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:27.869674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:27.868384    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:28.090506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:28.090472    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:28.422270 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:28.422226    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b94e65d_aff5_45f9_a4eb_8a54a4eaf439.slice/crio-c8242c79ccc06da71a522e6a31c4520b17aff87dd6d811c53dda7b2e710e15da WatchSource:0}: Error finding container c8242c79ccc06da71a522e6a31c4520b17aff87dd6d811c53dda7b2e710e15da: Status 404 returned error can't find the container with id c8242c79ccc06da71a522e6a31c4520b17aff87dd6d811c53dda7b2e710e15da
Oct 30 17:41:28.423070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:28.423045    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"]
Oct 30 17:41:28.425199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:28.425164    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:28.425278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:28.425237    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:28.425278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:28.425270    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:28.503491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:28.503460    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" event={"ID":"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439","Type":"ContainerStarted","Data":"c8242c79ccc06da71a522e6a31c4520b17aff87dd6d811c53dda7b2e710e15da"}
Oct 30 17:41:29.020656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:29.020610    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:29.513575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:29.513534    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" event={"ID":"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439","Type":"ContainerStarted","Data":"f1baf28d949b9804f2b36ef43e248c529f846650008ca9d53f9037295b3cc6ae"}
Oct 30 17:41:29.539825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:29.539789    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:29.578656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:29.578580    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podStartSLOduration=2.578560304 podStartE2EDuration="2.578560304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:29.576725618 +0000 UTC m=+1695.338769431" watchObservedRunningTime="2026-10-30 17:41:29.578560304 +0000 UTC m=+1695.340604118"
Oct 30 17:41:29.706796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:29.705962    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:30.090868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:30.090833    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:30.098813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:30.098779    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:30.520838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:30.520753    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:30.521878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:30.521859    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:41:31.199386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.199346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:31.342376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.342321    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:31.470349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.470268    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:31.640547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.640507    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"]
Oct 30 17:41:31.686506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.686070    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:31.706508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.706471    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"]
Oct 30 17:41:31.714333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.714290    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-74762d\""
Oct 30 17:41:31.714590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.714472    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-74762d-dockercfg-6h7s6\""
Oct 30 17:41:31.795831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.786840    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddsd\" (UniqueName: \"kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd\") pod \"payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") " pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:31.795831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.787382    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") " pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:31.896670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.896151    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") " pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:31.896670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.896238    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddsd\" (UniqueName: \"kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd\") pod \"payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") " pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:31.922711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.920101    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") " pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:31.948515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:31.947743    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddsd\" (UniqueName: \"kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd\") pod \"payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") " pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:32.031198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.031162    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:32.345990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.345957    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"]
Oct 30 17:41:32.346786 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:32.346761    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ba9c34_8c81_4b68_90dc_baa751b5cb93.slice/crio-738af6d52cc88e04ad9ce02e51d10071e70b2900eb01b510c24fe99151778034 WatchSource:0}: Error finding container 738af6d52cc88e04ad9ce02e51d10071e70b2900eb01b510c24fe99151778034: Status 404 returned error can't find the container with id 738af6d52cc88e04ad9ce02e51d10071e70b2900eb01b510c24fe99151778034
Oct 30 17:41:32.399349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.399313    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:32.517671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.517610    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:32.540078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.540046    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4" event={"ID":"78ba9c34-8c81-4b68-90dc-baa751b5cb93","Type":"ContainerStarted","Data":"5d732808a209c23ace4b4acfc62d13ab94ca79b5cfa8be3ae7dca9a81f538677"}
Oct 30 17:41:32.540288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.540109    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4" event={"ID":"78ba9c34-8c81-4b68-90dc-baa751b5cb93","Type":"ContainerStarted","Data":"738af6d52cc88e04ad9ce02e51d10071e70b2900eb01b510c24fe99151778034"}
Oct 30 17:41:32.540288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.540144    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:32.580741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:32.580689    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4" podStartSLOduration=1.5806703560000002 podStartE2EDuration="1.580670356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:32.579974882 +0000 UTC m=+1698.342018710" watchObservedRunningTime="2026-10-30 17:41:32.580670356 +0000 UTC m=+1698.342714170"
Oct 30 17:41:33.246148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:33.246118    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:34.228379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.228349    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:34.437022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.436992    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"]
Oct 30 17:41:34.441712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.441694    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.469393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.469361    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:34.470276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.470252    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bba90b-dockercfg-mkfmb\""
Oct 30 17:41:34.470428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.470260    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bba90b\""
Oct 30 17:41:34.475386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.475360    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"]
Oct 30 17:41:34.504330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.504249    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"]
Oct 30 17:41:34.510248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.510227    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:34.553674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.553173    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5nb\" (UniqueName: \"kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb\") pod \"payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.553674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.553470    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.665690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.660257    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.708692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.678627    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.708692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.660461    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5nb\" (UniqueName: \"kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb\") pod \"payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.708692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.681039    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42sh\" (UniqueName: \"kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh\") pod \"payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:34.708692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.681116    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:34.747675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.743119    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"]
Oct 30 17:41:34.747675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.744071    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5nb\" (UniqueName: \"kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb\") pod \"payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.763792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.758151    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:34.846019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.844110    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42sh\" (UniqueName: \"kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh\") pod \"payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:34.846019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.844202    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:34.846019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.845378    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:34.896714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:34.894136    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42sh\" (UniqueName: \"kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh\") pod \"payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:35.154671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.149956    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:35.326677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.324856    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"]
Oct 30 17:41:35.631004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.630963    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:35.659121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.659086    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" event={"ID":"26846168-75c8-4ec1-a92d-cfaec09a86a6","Type":"ContainerStarted","Data":"8bcfcae8f4215a3542535fd8a211ab92eeb5553cf100bc7ab3246fa909364af0"}
Oct 30 17:41:35.659314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.659173    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" event={"ID":"26846168-75c8-4ec1-a92d-cfaec09a86a6","Type":"ContainerStarted","Data":"bcdeba32f8910152348385c8d780aacfda56068b58e220b0ee67cb25f55d7874"}
Oct 30 17:41:35.659314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.659214    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:35.705199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.705105    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"]
Oct 30 17:41:35.710903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:35.710841    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" podStartSLOduration=1.710818436 podStartE2EDuration="1.710818436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:35.700745739 +0000 UTC m=+1701.462789574" watchObservedRunningTime="2026-10-30 17:41:35.710818436 +0000 UTC m=+1701.472862265"
Oct 30 17:41:35.715875 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:35.715843    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a4a0e3_162a_4e61_a7dd_e6e7d54350dd.slice/crio-237dde49d6093c0f2c8f9d6c2d553d56e072ea01645a26f0e13751d5832f5f2f WatchSource:0}: Error finding container 237dde49d6093c0f2c8f9d6c2d553d56e072ea01645a26f0e13751d5832f5f2f: Status 404 returned error can't find the container with id 237dde49d6093c0f2c8f9d6c2d553d56e072ea01645a26f0e13751d5832f5f2f
Oct 30 17:41:36.672680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:36.670344    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" event={"ID":"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd","Type":"ContainerStarted","Data":"340eb9b2c980441cd0142e4f382f0f3ed120278fd41134219b4d5ffa4390b2ea"}
Oct 30 17:41:36.672680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:36.670415    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" event={"ID":"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd","Type":"ContainerStarted","Data":"237dde49d6093c0f2c8f9d6c2d553d56e072ea01645a26f0e13751d5832f5f2f"}
Oct 30 17:41:36.672680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:36.670599    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:36.715693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:36.710858    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" podStartSLOduration=2.710835555 podStartE2EDuration="2.710835555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:36.709716349 +0000 UTC m=+1702.471760166" watchObservedRunningTime="2026-10-30 17:41:36.710835555 +0000 UTC m=+1702.472879371"
Oct 30 17:41:36.991104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:36.990985    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:37.748132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:37.748097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:38.225761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.225731    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"]
Oct 30 17:41:38.232481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.232459    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.251529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.251499    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"]
Oct 30 17:41:38.268670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.264958    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:41:38.268670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.265049    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:41:38.337042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.337011    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.337266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.337124    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.337266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.337164    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg9mz\" (UniqueName: \"kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.438291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.438250    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.438499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.438406    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.438499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.438490    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9mz\" (UniqueName: \"kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.439049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.439026    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.441116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.441090    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.489896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.489771    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9mz\" (UniqueName: \"kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz\") pod \"maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.547544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.547511    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:38.548552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.548529    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:38.999181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:38.999143    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"]
Oct 30 17:41:39.002098 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:39.002062    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57ab26d_0055_4a0e_972d_e11c0a92839f.slice/crio-eaa685070c0899fcc7bb814839a01e81e9579c9ce95d87afed72cac9e600fc6c WatchSource:0}: Error finding container eaa685070c0899fcc7bb814839a01e81e9579c9ce95d87afed72cac9e600fc6c: Status 404 returned error can't find the container with id eaa685070c0899fcc7bb814839a01e81e9579c9ce95d87afed72cac9e600fc6c
Oct 30 17:41:39.724283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:39.724241    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj" event={"ID":"f57ab26d-0055-4a0e-972d-e11c0a92839f","Type":"ContainerStarted","Data":"93528094050fb9782d0522e939e105162542c6721cf490cafdcd58ee52608c47"}
Oct 30 17:41:39.724498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:39.724303    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj" event={"ID":"f57ab26d-0055-4a0e-972d-e11c0a92839f","Type":"ContainerStarted","Data":"eaa685070c0899fcc7bb814839a01e81e9579c9ce95d87afed72cac9e600fc6c"}
Oct 30 17:41:39.778666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:39.777783    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj" podStartSLOduration=1.777762229 podStartE2EDuration="1.777762229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:39.777063174 +0000 UTC m=+1705.539106988" watchObservedRunningTime="2026-10-30 17:41:39.777762229 +0000 UTC m=+1705.539806043"
Oct 30 17:41:39.787518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:39.784857    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:40.401899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.401871    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"]
Oct 30 17:41:40.402356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.402049    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy" containerID="cri-o://a3945924bc7eac34e769d5d42c66ef54f3e3e81cf36db47b21bb1f182e60d7c0" gracePeriod=30
Oct 30 17:41:40.412880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.412845    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"]
Oct 30 17:41:40.413093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.413072    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" containerID="cri-o://810aa5e39d79d2e13754904c347f17694bee04c8a9c693cea0d04841912b3e9d" gracePeriod=30
Oct 30 17:41:40.418573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.418547    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"]
Oct 30 17:41:40.418806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.418781    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" containerID="cri-o://b273887164b2133519e5d4837016458a8cbd8ac43d7a0866d5b081fe4fa44759" gracePeriod=30
Oct 30 17:41:40.426811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.426782    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"]
Oct 30 17:41:40.427039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.427011    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy" containerID="cri-o://468b948e172e0ca47d8e72da98e066106e43121e920bc70fb59da3ce67bf9b4f" gracePeriod=30
Oct 30 17:41:40.735036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.734992    5534 generic.go:356] "Generic (PLEG): container finished" podID="f57ab26d-0055-4a0e-972d-e11c0a92839f" containerID="93528094050fb9782d0522e939e105162542c6721cf490cafdcd58ee52608c47" exitCode=0
Oct 30 17:41:40.735218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:40.735098    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj" event={"ID":"f57ab26d-0055-4a0e-972d-e11c0a92839f","Type":"ContainerDied","Data":"93528094050fb9782d0522e939e105162542c6721cf490cafdcd58ee52608c47"}
Oct 30 17:41:41.744066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:41.744016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:41.922661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:41.911431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:42.132949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.128265    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:42.493692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.489921    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:42.624412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.623656    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca\") pod \"f57ab26d-0055-4a0e-972d-e11c0a92839f\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") "
Oct 30 17:41:42.624412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.623985    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz\" (UniqueName: \"kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz\") pod \"f57ab26d-0055-4a0e-972d-e11c0a92839f\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") "
Oct 30 17:41:42.624412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.624010    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca\") pod \"f57ab26d-0055-4a0e-972d-e11c0a92839f\" (UID: \"f57ab26d-0055-4a0e-972d-e11c0a92839f\") "
Oct 30 17:41:42.624738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.624566    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca" pod "f57ab26d-0055-4a0e-972d-e11c0a92839f" (UID: "f57ab26d-0055-4a0e-972d-e11c0a92839f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:42.624896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.624834    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:42.627529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.627504    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz" pod "f57ab26d-0055-4a0e-972d-e11c0a92839f" (UID: "f57ab26d-0055-4a0e-972d-e11c0a92839f"). InnerVolumeSpecName "kube-api-access-rg9mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:42.631147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.631114    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca" pod "f57ab26d-0055-4a0e-972d-e11c0a92839f" (UID: "f57ab26d-0055-4a0e-972d-e11c0a92839f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:42.726262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.726165    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg9mz\" (UniqueName: \"kubernetes.io/projected/f57ab26d-0055-4a0e-972d-e11c0a92839f-kube-api-access-rg9mz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:42.726262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.726204    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57ab26d-0055-4a0e-972d-e11c0a92839f-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:42.777669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.774924    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj" event={"ID":"f57ab26d-0055-4a0e-972d-e11c0a92839f","Type":"ContainerDied","Data":"eaa685070c0899fcc7bb814839a01e81e9579c9ce95d87afed72cac9e600fc6c"}
Oct 30 17:41:42.777669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.774990    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaa685070c0899fcc7bb814839a01e81e9579c9ce95d87afed72cac9e600fc6c"
Oct 30 17:41:42.777669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:42.775065    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"
Oct 30 17:41:43.553756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:43.553670    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:41:44.003589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:44.003548    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:45.059285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.059252    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:45.253743 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:41:45.253700    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:41:45.402614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.402519    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:45.680308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.679706    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:45.729258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.723595    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:41:45.729258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.723678    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:41:45.804981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.804945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:45.990958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.990406    5534 generic.go:356] "Generic (PLEG): container finished" podID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerID="810aa5e39d79d2e13754904c347f17694bee04c8a9c693cea0d04841912b3e9d" exitCode=0
Oct 30 17:41:45.990958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.990685    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" event={"ID":"683cd04d-90d2-4f93-b86d-c72d0f33bf13","Type":"ContainerDied","Data":"810aa5e39d79d2e13754904c347f17694bee04c8a9c693cea0d04841912b3e9d"}
Oct 30 17:41:45.990958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.990752    5534 scope.go:122] "RemoveContainer" containerID="2a5e06a4089631dff13a86b92f407ed5790e6cf7a085c4dcc7b69e64b5af5ade"
Oct 30 17:41:45.991698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.991337    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:41:45.997899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.997498    5534 generic.go:356] "Generic (PLEG): container finished" podID="6a9447b2-7531-4460-9c14-4888f084ad65" containerID="a3945924bc7eac34e769d5d42c66ef54f3e3e81cf36db47b21bb1f182e60d7c0" exitCode=0
Oct 30 17:41:45.997899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:45.997676    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" event={"ID":"6a9447b2-7531-4460-9c14-4888f084ad65","Type":"ContainerDied","Data":"a3945924bc7eac34e769d5d42c66ef54f3e3e81cf36db47b21bb1f182e60d7c0"}
Oct 30 17:41:46.031728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.023725    5534 generic.go:356] "Generic (PLEG): container finished" podID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerID="b273887164b2133519e5d4837016458a8cbd8ac43d7a0866d5b081fe4fa44759" exitCode=0
Oct 30 17:41:46.031728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.023916    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" event={"ID":"6946f94e-959d-46c4-a9c9-52b17b1cb7c9","Type":"ContainerDied","Data":"b273887164b2133519e5d4837016458a8cbd8ac43d7a0866d5b081fe4fa44759"}
Oct 30 17:41:46.035394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.034792    5534 generic.go:356] "Generic (PLEG): container finished" podID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerID="468b948e172e0ca47d8e72da98e066106e43121e920bc70fb59da3ce67bf9b4f" exitCode=0
Oct 30 17:41:46.035394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.034851    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff" event={"ID":"ab071bf4-e05f-499b-bb67-9d83c97cd71d","Type":"ContainerDied","Data":"468b948e172e0ca47d8e72da98e066106e43121e920bc70fb59da3ce67bf9b4f"}
Oct 30 17:41:46.035394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.035185    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"
Oct 30 17:41:46.038378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.038281    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:41:46.047973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.047874    5534 scope.go:122] "RemoveContainer" containerID="468b948e172e0ca47d8e72da98e066106e43121e920bc70fb59da3ce67bf9b4f"
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.154936    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155072    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155146    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155176    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155217    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155267    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155310    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155429    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155483    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155539    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155570    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155600    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155669    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155734    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155772    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155816    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.155902    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token\") pod \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\" (UID: \"683cd04d-90d2-4f93-b86d-c72d0f33bf13\") "
Oct 30 17:41:46.156499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.156009    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert\") pod \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\" (UID: \"ab071bf4-e05f-499b-bb67-9d83c97cd71d\") "
Oct 30 17:41:46.161738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.161224    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.161738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.161434    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.162271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.162119    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:46.163209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.162849    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.163411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.163378    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.166323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.164453    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.166991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.166955    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.167390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.167239    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:46.167390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.167305    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.167390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.167362    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.171458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.171411    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "kube-api-access-wff2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.173957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.173912    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.174136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.174084    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:46.174441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.174395    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.174965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.174628    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.178934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.178061    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "kube-api-access-47pd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.178934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.178188    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy" pod "ab071bf4-e05f-499b-bb67-9d83c97cd71d" (UID: "ab071bf4-e05f-499b-bb67-9d83c97cd71d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.186247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.186209    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo" pod "683cd04d-90d2-4f93-b86d-c72d0f33bf13" (UID: "683cd04d-90d2-4f93-b86d-c72d0f33bf13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:46.249883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.249578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262060    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262089    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262103    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262115    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262127    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262140    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262155    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wff2q\" (UniqueName: \"kubernetes.io/projected/683cd04d-90d2-4f93-b86d-c72d0f33bf13-kube-api-access-wff2q\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262168    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262181    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262193    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262206    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262218    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262230    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47pd6\" (UniqueName: \"kubernetes.io/projected/ab071bf4-e05f-499b-bb67-9d83c97cd71d-kube-api-access-47pd6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262244    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262256    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262268    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262282    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab071bf4-e05f-499b-bb67-9d83c97cd71d-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.262316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.262295    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683cd04d-90d2-4f93-b86d-c72d0f33bf13-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.347308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.346869    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:41:46.408699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.408212    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:41:46.467239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.467206    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.467568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.467529    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.467802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.467677    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.467943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.467915    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.467943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.467917    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468004    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468173    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468255    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468366    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468439    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468670    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy\") pod \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\" (UID: \"6946f94e-959d-46c4-a9c9-52b17b1cb7c9\") "
Oct 30 17:41:46.468951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.468914    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:46.470311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.470027    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.470311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.470053    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.470311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.470070    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.470907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.470885    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.471032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.470909    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.478182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.478158    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:46.478328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.478310    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "kube-api-access-sx8tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.478412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.478395    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.481370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.481348    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token" pod "6946f94e-959d-46c4-a9c9-52b17b1cb7c9" (UID: "6946f94e-959d-46c4-a9c9-52b17b1cb7c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.486987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.486965    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"]
Oct 30 17:41:46.499282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.499258    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-ww2ff"]
Oct 30 17:41:46.572073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.571624    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.571788    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.571829    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.571859    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.571946    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.571981    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.572101    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.572156    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.572235    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token\") pod \"6a9447b2-7531-4460-9c14-4888f084ad65\" (UID: \"6a9447b2-7531-4460-9c14-4888f084ad65\") "
Oct 30 17:41:46.572772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.572269    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573071    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8tg\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-kube-api-access-sx8tg\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573098    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573113    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573126    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573176    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573188    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573201    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6946f94e-959d-46c4-a9c9-52b17b1cb7c9-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573943    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573954    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.574007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.573970    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.575028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.574989    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:46.575451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.575424    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "kube-api-access-ln6kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.589819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.589762    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:46.591823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.591772    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:46.603348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.603289    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy" pod "6a9447b2-7531-4460-9c14-4888f084ad65" (UID: "6a9447b2-7531-4460-9c14-4888f084ad65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:46.674414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674381    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9447b2-7531-4460-9c14-4888f084ad65-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674412    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674422    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674431    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6kj\" (UniqueName: \"kubernetes.io/projected/6a9447b2-7531-4460-9c14-4888f084ad65-kube-api-access-ln6kj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674447    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674457    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674465    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9447b2-7531-4460-9c14-4888f084ad65-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.674737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.674473    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9447b2-7531-4460-9c14-4888f084ad65-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:46.676035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.676010    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:41:46.947120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.947082    5534 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:41:46.947309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:46.947163    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:41:47.009258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.009229    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:47.047174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.047136    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"
Oct 30 17:41:47.047174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.047156    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74" event={"ID":"683cd04d-90d2-4f93-b86d-c72d0f33bf13","Type":"ContainerDied","Data":"ca4e439b14bf35520389d600717477fe451ba9bcaea93041467cc0c9a271c9e1"}
Oct 30 17:41:47.047402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.047224    5534 scope.go:122] "RemoveContainer" containerID="810aa5e39d79d2e13754904c347f17694bee04c8a9c693cea0d04841912b3e9d"
Oct 30 17:41:47.049093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.049077    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"
Oct 30 17:41:47.049357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.049086    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd" event={"ID":"6a9447b2-7531-4460-9c14-4888f084ad65","Type":"ContainerDied","Data":"d60a31dad9c44fe5d3b9c3e457fef7844e0471b27dafb8e99214823c8c3edb5d"}
Oct 30 17:41:47.051742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.051714    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp" event={"ID":"6946f94e-959d-46c4-a9c9-52b17b1cb7c9","Type":"ContainerDied","Data":"eb6a24af527ceb58926fc10e433912dab8a1274c4847e53ba920fee9123ed344"}
Oct 30 17:41:47.051871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.051821    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"
Oct 30 17:41:47.060817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.060798    5534 scope.go:122] "RemoveContainer" containerID="a3945924bc7eac34e769d5d42c66ef54f3e3e81cf36db47b21bb1f182e60d7c0"
Oct 30 17:41:47.073252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.073226    5534 scope.go:122] "RemoveContainer" containerID="b273887164b2133519e5d4837016458a8cbd8ac43d7a0866d5b081fe4fa44759"
Oct 30 17:41:47.163376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.163335    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"]
Oct 30 17:41:47.167581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.167553    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-xfdcd"]
Oct 30 17:41:47.199334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.199248    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"]
Oct 30 17:41:47.203103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.203076    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dfwkp"]
Oct 30 17:41:47.245269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.245232    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"]
Oct 30 17:41:47.249801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.249774    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-tzz74"]
Oct 30 17:41:47.573342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.573262    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:47.653328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.653298    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"]
Oct 30 17:41:47.654290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654268    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy"
Oct 30 17:41:47.654290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654290    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654318    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f57ab26d-0055-4a0e-972d-e11c0a92839f" containerName="revoke-keys"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654327    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ab26d-0055-4a0e-972d-e11c0a92839f" containerName="revoke-keys"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654378    5534 cpu_manager.go:414] "containerMap: removing container" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654386    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654413    5534 cpu_manager.go:414] "containerMap: removing container" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654420    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654435    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654443    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654453    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy"
Oct 30 17:41:47.654605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654460    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy"
Oct 30 17:41:47.655463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654707    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ab26d-0055-4a0e-972d-e11c0a92839f" containerName="revoke-keys"
Oct 30 17:41:47.655463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654735    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy"
Oct 30 17:41:47.655463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654748    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" containerName="istio-proxy"
Oct 30 17:41:47.655463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654757    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" containerName="istio-proxy"
Oct 30 17:41:47.655463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.654774    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" containerName="istio-proxy"
Oct 30 17:41:47.660106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.660081    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:47.679230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.679205    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-38f445-dockercfg-jsxsq\""
Oct 30 17:41:47.686592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.686563    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"]
Oct 30 17:41:47.690770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.690736    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w7l\" (UniqueName: \"kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l\") pod \"payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:47.690914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.690811    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:47.695803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.695784    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:41:47.789789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.733990    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-38f445\""
Oct 30 17:41:47.892990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.892954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:47.909881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.904264    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w7l\" (UniqueName: \"kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l\") pod \"payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:47.909881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.907914    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:47.909881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.909810    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:48.000859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:47.994994    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w7l\" (UniqueName: \"kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l\") pod \"payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:48.020751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.020720    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"]
Oct 30 17:41:48.022731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.022708    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" containerName="istio-proxy"
Oct 30 17:41:48.034493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.033716    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.037846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.033605    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"]
Oct 30 17:41:48.213950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.213864    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9khx\" (UniqueName: \"kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx\") pod \"payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.214452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.214202    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.292185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.292154    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:48.316724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.316386    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9khx\" (UniqueName: \"kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx\") pod \"payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.316724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.316636    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.317493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.317468    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.343914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.343847    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9khx\" (UniqueName: \"kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx\") pod \"payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.351921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.351893    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:48.376807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.376785    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683cd04d-90d2-4f93-b86d-c72d0f33bf13" path="/var/lib/kubelet/pods/683cd04d-90d2-4f93-b86d-c72d0f33bf13/volumes"
Oct 30 17:41:48.377735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.377421    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6946f94e-959d-46c4-a9c9-52b17b1cb7c9" path="/var/lib/kubelet/pods/6946f94e-959d-46c4-a9c9-52b17b1cb7c9/volumes"
Oct 30 17:41:48.378279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.378262    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9447b2-7531-4460-9c14-4888f084ad65" path="/var/lib/kubelet/pods/6a9447b2-7531-4460-9c14-4888f084ad65/volumes"
Oct 30 17:41:48.379463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.379415    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab071bf4-e05f-499b-bb67-9d83c97cd71d" path="/var/lib/kubelet/pods/ab071bf4-e05f-499b-bb67-9d83c97cd71d/volumes"
Oct 30 17:41:48.618717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.618438    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"]
Oct 30 17:41:48.645783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.645754    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:48.712222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.712196    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"]
Oct 30 17:41:48.906025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:48.905938    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:49.038832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.038763    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:49.079988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.079938    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" event={"ID":"8e12ba46-6722-4e2f-9369-895b292d605c","Type":"ContainerStarted","Data":"2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95"}
Oct 30 17:41:49.080293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.080269    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:41:49.080486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.080462    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" event={"ID":"8e12ba46-6722-4e2f-9369-895b292d605c","Type":"ContainerStarted","Data":"5be90bd95875526d6be029430baf91f50890f76c950aabbb93d0ec5eaf0064ef"}
Oct 30 17:41:49.082610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.082578    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" event={"ID":"4cbd05e7-3658-4053-9901-0b1383bfc596","Type":"ContainerStarted","Data":"24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08"}
Oct 30 17:41:49.084278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.082669    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" event={"ID":"4cbd05e7-3658-4053-9901-0b1383bfc596","Type":"ContainerStarted","Data":"e122fd15d40c5c1bc063d4421fac049617111f6acae72c5ce6d51af22081d51d"}
Oct 30 17:41:49.084278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.082728    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:41:49.088969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.088771    5534 generic.go:356] "Generic (PLEG): container finished" podID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerID="ff695258dd354590231a26c6c6e6a8130ed7971cd2d60abb5639d2e785887f4d" exitCode=0
Oct 30 17:41:49.089117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.089027    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" event={"ID":"af1f1ddd-0676-4413-9dd2-1ad060ec5090","Type":"ContainerDied","Data":"ff695258dd354590231a26c6c6e6a8130ed7971cd2d60abb5639d2e785887f4d"}
Oct 30 17:41:49.089625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.089607    5534 scope.go:122] "RemoveContainer" containerID="ff695258dd354590231a26c6c6e6a8130ed7971cd2d60abb5639d2e785887f4d"
Oct 30 17:41:49.091596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.091565    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:49.091737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.091690    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:49.091737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.091732    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:41:49.121409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.121355    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" podStartSLOduration=2.121336271 podStartE2EDuration="2.121336271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:49.120834337 +0000 UTC m=+1714.882878276" watchObservedRunningTime="2026-10-30 17:41:49.121336271 +0000 UTC m=+1714.883380085"
Oct 30 17:41:49.216190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:49.216119    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" podStartSLOduration=2.216097295 podStartE2EDuration="2.216097295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:49.208607467 +0000 UTC m=+1714.970651283" watchObservedRunningTime="2026-10-30 17:41:49.216097295 +0000 UTC m=+1714.978141110"
Oct 30 17:41:50.099060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:50.099026    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" event={"ID":"af1f1ddd-0676-4413-9dd2-1ad060ec5090","Type":"ContainerStarted","Data":"f339d098e6641e3f7fde6fa66bf1c76fcca481dfb19228b0c85ac93196a8b196"}
Oct 30 17:41:50.393422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:50.393335    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:41:50.406040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:50.405989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:51.011271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:51.011237    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:41:51.012263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:51.012240    5534 patch_prober.go:29] interesting pod/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 17:41:51.012373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:51.012296    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 17:41:52.011993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:52.011959    5534 patch_prober.go:29] interesting pod/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 17:41:52.012426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:52.012016    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 17:41:52.197775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:52.197745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:53.011197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:53.011161    5534 patch_prober.go:29] interesting pod/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 17:41:53.011392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:53.011218    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 17:41:53.575191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:53.575156    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:54.011675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.011472    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"]
Oct 30 17:41:54.011888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.011774    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy" containerID="cri-o://8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c" gracePeriod=30
Oct 30 17:41:54.018010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.017986    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"]
Oct 30 17:41:54.018390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.018361    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy" containerID="cri-o://3413cbe5e0a3095396d6c188dc2b9ee0234f1134fa418a7c3090bbd955b5906c" gracePeriod=30
Oct 30 17:41:54.026881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.023952    5534 patch_prober.go:29] interesting pod/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 17:41:54.026881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.023998    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 17:41:54.026881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.026514    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"]
Oct 30 17:41:54.026881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:54.026776    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy" containerID="cri-o://f6d924aa61376269ba9d007f717ca66ddba7b1d2289a35a3a1dc2d7bb2dab684" gracePeriod=30
Oct 30 17:41:55.011628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.011550    5534 patch_prober.go:29] interesting pod/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 17:41:55.011628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.011603    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 17:41:55.321207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.321119    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:55.617791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.617751    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"]
Oct 30 17:41:55.622421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.622401    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.639251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.639221    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:41:55.644113 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.644081    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"]
Oct 30 17:41:55.703384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.703344    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7sw2\" (UniqueName: \"kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.703585 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.703419    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.703682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.703664    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.738328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.738298    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:41:55.804894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.804855    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.805118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.805022    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sw2\" (UniqueName: \"kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.805118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.805057    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.805770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.805750    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.807514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.807491    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.828823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.828793    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7sw2\" (UniqueName: \"kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2\") pod \"maas-api-revoke-sub-af79b3ae-4r6zc\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:55.939862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:55.939770    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:56.029803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:56.029759    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:41:56.032900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:56.032862    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:41:56.232493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:56.232400    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:56.256036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:56.256000    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"]
Oct 30 17:41:56.256657 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:41:56.256613    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode017f70d_c0a3_4ad3_a567_9d5576fcc3be.slice/crio-dc853b193808405299996850c85245bee885d78b0be9c58d0c4b082e0b415825 WatchSource:0}: Error finding container dc853b193808405299996850c85245bee885d78b0be9c58d0c4b082e0b415825: Status 404 returned error can't find the container with id dc853b193808405299996850c85245bee885d78b0be9c58d0c4b082e0b415825
Oct 30 17:41:56.296069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:56.296026    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc" event={"ID":"e017f70d-c0a3-4ad3-a567-9d5576fcc3be","Type":"ContainerStarted","Data":"dc853b193808405299996850c85245bee885d78b0be9c58d0c4b082e0b415825"}
Oct 30 17:41:57.307538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:57.307499    5534 generic.go:356] "Generic (PLEG): container finished" podID="e017f70d-c0a3-4ad3-a567-9d5576fcc3be" containerID="9b43f3e6e7a1feea010602113a34fc62475aa3974cf5fff0e4afdc66c32f2dfa" exitCode=0
Oct 30 17:41:57.308045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:57.307565    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc" event={"ID":"e017f70d-c0a3-4ad3-a567-9d5576fcc3be","Type":"ContainerDied","Data":"9b43f3e6e7a1feea010602113a34fc62475aa3974cf5fff0e4afdc66c32f2dfa"}
Oct 30 17:41:57.992816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:57.992776    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:41:58.501265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.501235    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:58.552605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.552567    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2\" (UniqueName: \"kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2\") pod \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") "
Oct 30 17:41:58.557860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.556800    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca\") pod \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") "
Oct 30 17:41:58.557860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.556911    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca\") pod \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\" (UID: \"e017f70d-c0a3-4ad3-a567-9d5576fcc3be\") "
Oct 30 17:41:58.559742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.558283    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca" pod "e017f70d-c0a3-4ad3-a567-9d5576fcc3be" (UID: "e017f70d-c0a3-4ad3-a567-9d5576fcc3be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:58.561451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.561309    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:58.580950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.580907    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2" pod "e017f70d-c0a3-4ad3-a567-9d5576fcc3be" (UID: "e017f70d-c0a3-4ad3-a567-9d5576fcc3be"). InnerVolumeSpecName "kube-api-access-z7sw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:58.588094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.588063    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca" pod "e017f70d-c0a3-4ad3-a567-9d5576fcc3be" (UID: "e017f70d-c0a3-4ad3-a567-9d5576fcc3be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:58.663139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.663103    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7sw2\" (UniqueName: \"kubernetes.io/projected/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-kube-api-access-z7sw2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:58.663139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:58.663142    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017f70d-c0a3-4ad3-a567-9d5576fcc3be-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.314717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.314692    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:59.332184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.332019    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc" event={"ID":"e017f70d-c0a3-4ad3-a567-9d5576fcc3be","Type":"ContainerDied","Data":"dc853b193808405299996850c85245bee885d78b0be9c58d0c4b082e0b415825"}
Oct 30 17:41:59.332184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.332115    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc853b193808405299996850c85245bee885d78b0be9c58d0c4b082e0b415825"
Oct 30 17:41:59.332184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.332114    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"
Oct 30 17:41:59.334615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.334591    5534 generic.go:356] "Generic (PLEG): container finished" podID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerID="f6d924aa61376269ba9d007f717ca66ddba7b1d2289a35a3a1dc2d7bb2dab684" exitCode=0
Oct 30 17:41:59.334846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.334760    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" event={"ID":"2d80f646-adfc-4423-9d74-af2f9ef75990","Type":"ContainerDied","Data":"f6d924aa61376269ba9d007f717ca66ddba7b1d2289a35a3a1dc2d7bb2dab684"}
Oct 30 17:41:59.336691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.336631    5534 generic.go:356] "Generic (PLEG): container finished" podID="b31f0143-e325-44c1-9803-af5c1708e079" containerID="3413cbe5e0a3095396d6c188dc2b9ee0234f1134fa418a7c3090bbd955b5906c" exitCode=0
Oct 30 17:41:59.336811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.336748    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" event={"ID":"b31f0143-e325-44c1-9803-af5c1708e079","Type":"ContainerDied","Data":"3413cbe5e0a3095396d6c188dc2b9ee0234f1134fa418a7c3090bbd955b5906c"}
Oct 30 17:41:59.338689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.338582    5534 generic.go:356] "Generic (PLEG): container finished" podID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerID="8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c" exitCode=0
Oct 30 17:41:59.338843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.338694    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" event={"ID":"f5ff310f-c0ac-4b45-b91c-1433f37caa27","Type":"ContainerDied","Data":"8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c"}
Oct 30 17:41:59.338843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.338733    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx" event={"ID":"f5ff310f-c0ac-4b45-b91c-1433f37caa27","Type":"ContainerDied","Data":"b09415531cc700c8e95cf0ea367a54dbf4fa8de719c43250a4e87b00c123ad44"}
Oct 30 17:41:59.338843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.338764    5534 scope.go:122] "RemoveContainer" containerID="8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c"
Oct 30 17:41:59.339044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.338916    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"
Oct 30 17:41:59.358747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.357686    5534 scope.go:122] "RemoveContainer" containerID="8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c"
Oct 30 17:41:59.358747 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:41:59.358041    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c\": container with ID starting with 8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c not found: ID does not exist" containerID="8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c"
Oct 30 17:41:59.358747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.358078    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c"} err="failed to get container status \"8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c\": rpc error: code = NotFound desc = could not find container \"8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c\": container with ID starting with 8a29803970b204906693ef4c2e40af29fdc08ea93ce16d07fa2521cfdf62fe4c not found: ID does not exist"
Oct 30 17:41:59.377679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.377368    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.377870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.377844    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.377939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.377897    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.378000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.377954    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.383427    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384147    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384412    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384512    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384573    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384611    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384705    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert\") pod \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\" (UID: \"f5ff310f-c0ac-4b45-b91c-1433f37caa27\") "
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384906    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.384979    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.385301    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:59.389689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.387154    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:59.390568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.390186    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "kube-api-access-dpphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:59.392181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392157    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.392181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392175    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.392181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392186    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.392406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392194    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.392406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392203    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.392406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392211    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpphp\" (UniqueName: \"kubernetes.io/projected/f5ff310f-c0ac-4b45-b91c-1433f37caa27-kube-api-access-dpphp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.392406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.392219    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.394932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.393543    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:59.394932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.394549    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy" pod "f5ff310f-c0ac-4b45-b91c-1433f37caa27" (UID: "f5ff310f-c0ac-4b45-b91c-1433f37caa27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.407116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.407089    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:41:59.440737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.440707    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.517722    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.517791    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.517833    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.517912    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518157    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518235    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518287    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518320    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518379    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket\") pod \"b31f0143-e325-44c1-9803-af5c1708e079\" (UID: \"b31f0143-e325-44c1-9803-af5c1708e079\") "
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518550    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.519274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.518855    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.520476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.519360    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.520476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.519384    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.520476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.519399    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.520476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.519414    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ff310f-c0ac-4b45-b91c-1433f37caa27-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.524589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.524562    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "kube-api-access-f6ssp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:59.524937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.524916    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:59.526054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.525477    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.526211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.526187    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:59.527694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.527667    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.530122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.530091    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:59.530896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.530766    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy" pod "b31f0143-e325-44c1-9803-af5c1708e079" (UID: "b31f0143-e325-44c1-9803-af5c1708e079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620265    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620318    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620376    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620432    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620581    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620605    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620731    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620763    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.620796    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket\") pod \"2d80f646-adfc-4423-9d74-af2f9ef75990\" (UID: \"2d80f646-adfc-4423-9d74-af2f9ef75990\") "
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621302    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621323    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621339    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621353    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f0143-e325-44c1-9803-af5c1708e079-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621364    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f0143-e325-44c1-9803-af5c1708e079-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621377    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ssp\" (UniqueName: \"kubernetes.io/projected/b31f0143-e325-44c1-9803-af5c1708e079-kube-api-access-f6ssp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621386    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f0143-e325-44c1-9803-af5c1708e079-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.621591    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.622166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.622132    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.624061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.622535    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.624061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.622699    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:59.624061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.622765    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.626135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.626099    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:59.626135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.626115    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:59.626291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.626212    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "kube-api-access-l2845". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:59.626915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.626892    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token" pod "2d80f646-adfc-4423-9d74-af2f9ef75990" (UID: "2d80f646-adfc-4423-9d74-af2f9ef75990"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:59.708636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.708478    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"]
Oct 30 17:41:59.712227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.712203    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-jcqbx"]
Oct 30 17:41:59.723157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723123    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723152    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723163    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723175    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723184    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723192    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2845\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-kube-api-access-l2845\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723201    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d80f646-adfc-4423-9d74-af2f9ef75990-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723211    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d80f646-adfc-4423-9d74-af2f9ef75990-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.723345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.723218    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d80f646-adfc-4423-9d74-af2f9ef75990-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:41:59.876499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.876402    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"]
Oct 30 17:41:59.877608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877582    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy"
Oct 30 17:41:59.877608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877607    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy"
Oct 30 17:41:59.877839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877662    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy"
Oct 30 17:41:59.877839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877671    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy"
Oct 30 17:41:59.877839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877701    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy"
Oct 30 17:41:59.877839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877711    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy"
Oct 30 17:41:59.877839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877770    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e017f70d-c0a3-4ad3-a567-9d5576fcc3be" containerName="revoke-keys"
Oct 30 17:41:59.877839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877778    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e017f70d-c0a3-4ad3-a567-9d5576fcc3be" containerName="revoke-keys"
Oct 30 17:41:59.878209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877947    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" containerName="istio-proxy"
Oct 30 17:41:59.878209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877964    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" containerName="istio-proxy"
Oct 30 17:41:59.878209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877977    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31f0143-e325-44c1-9803-af5c1708e079" containerName="istio-proxy"
Oct 30 17:41:59.878209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.877993    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e017f70d-c0a3-4ad3-a567-9d5576fcc3be" containerName="revoke-keys"
Oct 30 17:41:59.885007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.884977    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:41:59.904966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.904932    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"]
Oct 30 17:41:59.905543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.905512    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:41:59.906612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.906590    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:41:59.934215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:41:59.934181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:00.028579 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.028533    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.028850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.028635    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.028975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.028925    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh96q\" (UniqueName: \"kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.104889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.104854    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:42:00.107364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.107302    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:42:00.130527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.130453    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.130527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.130498    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.130813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.130659    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh96q\" (UniqueName: \"kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.131186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.131163    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.133312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.133290    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.155255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.155224    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh96q\" (UniqueName: \"kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q\") pod \"maas-api-revoke-keys-e2e-api-a-802bda-2vbxw\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.216392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.216078    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:00.358795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.356693    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f" event={"ID":"2d80f646-adfc-4423-9d74-af2f9ef75990","Type":"ContainerDied","Data":"57090ab6b924c7b907a3fd4a834f3ad71dabe7e94beb4ea787fc0bb6a4c99cdb"}
Oct 30 17:42:00.358795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.356758    5534 scope.go:122] "RemoveContainer" containerID="f6d924aa61376269ba9d007f717ca66ddba7b1d2289a35a3a1dc2d7bb2dab684"
Oct 30 17:42:00.358795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.356846    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"
Oct 30 17:42:00.385655 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.385561    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"
Oct 30 17:42:00.388963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.388935    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ff310f-c0ac-4b45-b91c-1433f37caa27" path="/var/lib/kubelet/pods/f5ff310f-c0ac-4b45-b91c-1433f37caa27/volumes"
Oct 30 17:42:00.389931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.389886    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk" event={"ID":"b31f0143-e325-44c1-9803-af5c1708e079","Type":"ContainerDied","Data":"967f841580f1014326d25d8df3de5e376a734af76dd3e43bc69b1000d0bbdd90"}
Oct 30 17:42:00.390003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.389942    5534 scope.go:122] "RemoveContainer" containerID="3413cbe5e0a3095396d6c188dc2b9ee0234f1134fa418a7c3090bbd955b5906c"
Oct 30 17:42:00.425220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.425192    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"]
Oct 30 17:42:00.433179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.433149    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-cw26f"]
Oct 30 17:42:00.461878 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:42:00.461839    5534 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d80f646_adfc_4423_9d74_af2f9ef75990.slice/crio-57090ab6b924c7b907a3fd4a834f3ad71dabe7e94beb4ea787fc0bb6a4c99cdb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31f0143_e325_44c1_9803_af5c1708e079.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 17:42:00.476698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.476626    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"]
Oct 30 17:42:00.483553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.483522    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-9nddk"]
Oct 30 17:42:00.535557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:00.535526    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"]
Oct 30 17:42:00.537113 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:42:00.537082    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2de424_b8da_44c3_a954_5ca9e1545984.slice/crio-c28d1c404c0a18bbed7b557acbd88ab73ca3b1edf12148988fdfb44c5f2cfd5e WatchSource:0}: Error finding container c28d1c404c0a18bbed7b557acbd88ab73ca3b1edf12148988fdfb44c5f2cfd5e: Status 404 returned error can't find the container with id c28d1c404c0a18bbed7b557acbd88ab73ca3b1edf12148988fdfb44c5f2cfd5e
Oct 30 17:42:01.401532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:01.401495    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw" event={"ID":"4b2de424-b8da-44c3-a954-5ca9e1545984","Type":"ContainerStarted","Data":"17b5fcbda415327facd505d370e61e929140d1b6f2b868d44aa6b888c2a62f92"}
Oct 30 17:42:01.401774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:01.401560    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw" event={"ID":"4b2de424-b8da-44c3-a954-5ca9e1545984","Type":"ContainerStarted","Data":"c28d1c404c0a18bbed7b557acbd88ab73ca3b1edf12148988fdfb44c5f2cfd5e"}
Oct 30 17:42:01.441801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:01.441738    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw" podStartSLOduration=2.441718894 podStartE2EDuration="2.441718894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:01.440899822 +0000 UTC m=+1727.202943652" watchObservedRunningTime="2026-10-30 17:42:01.441718894 +0000 UTC m=+1727.203762709"
Oct 30 17:42:01.868046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:01.867280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:02.033304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:02.033269    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:02.373426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:02.373390    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d80f646-adfc-4423-9d74-af2f9ef75990" path="/var/lib/kubelet/pods/2d80f646-adfc-4423-9d74-af2f9ef75990/volumes"
Oct 30 17:42:02.374002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:02.373983    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31f0143-e325-44c1-9803-af5c1708e079" path="/var/lib/kubelet/pods/b31f0143-e325-44c1-9803-af5c1708e079/volumes"
Oct 30 17:42:02.413090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:02.413051    5534 generic.go:356] "Generic (PLEG): container finished" podID="4b2de424-b8da-44c3-a954-5ca9e1545984" containerID="17b5fcbda415327facd505d370e61e929140d1b6f2b868d44aa6b888c2a62f92" exitCode=0
Oct 30 17:42:02.413293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:02.413149    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw" event={"ID":"4b2de424-b8da-44c3-a954-5ca9e1545984","Type":"ContainerDied","Data":"17b5fcbda415327facd505d370e61e929140d1b6f2b868d44aa6b888c2a62f92"}
Oct 30 17:42:03.616800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.616771    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:03.688414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.688298    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca\") pod \"4b2de424-b8da-44c3-a954-5ca9e1545984\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") "
Oct 30 17:42:03.688613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.688468    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca\") pod \"4b2de424-b8da-44c3-a954-5ca9e1545984\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") "
Oct 30 17:42:03.688613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.688570    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q\" (UniqueName: \"kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q\") pod \"4b2de424-b8da-44c3-a954-5ca9e1545984\" (UID: \"4b2de424-b8da-44c3-a954-5ca9e1545984\") "
Oct 30 17:42:03.690068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.690034    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca" pod "4b2de424-b8da-44c3-a954-5ca9e1545984" (UID: "4b2de424-b8da-44c3-a954-5ca9e1545984"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:42:03.692291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.692255    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q" pod "4b2de424-b8da-44c3-a954-5ca9e1545984" (UID: "4b2de424-b8da-44c3-a954-5ca9e1545984"). InnerVolumeSpecName "kube-api-access-bh96q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:42:03.692755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.692713    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca" pod "4b2de424-b8da-44c3-a954-5ca9e1545984" (UID: "4b2de424-b8da-44c3-a954-5ca9e1545984"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:42:03.791561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.791522    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:03.791561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.791565    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2de424-b8da-44c3-a954-5ca9e1545984-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:03.791817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.791580    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh96q\" (UniqueName: \"kubernetes.io/projected/4b2de424-b8da-44c3-a954-5ca9e1545984-kube-api-access-bh96q\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:03.823467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:03.823434    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:04.431179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:04.431143    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw" event={"ID":"4b2de424-b8da-44c3-a954-5ca9e1545984","Type":"ContainerDied","Data":"c28d1c404c0a18bbed7b557acbd88ab73ca3b1edf12148988fdfb44c5f2cfd5e"}
Oct 30 17:42:04.431366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:04.431198    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c28d1c404c0a18bbed7b557acbd88ab73ca3b1edf12148988fdfb44c5f2cfd5e"
Oct 30 17:42:04.431366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:04.431159    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"
Oct 30 17:42:05.473577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.473538    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"]
Oct 30 17:42:05.474755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.474733    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4b2de424-b8da-44c3-a954-5ca9e1545984" containerName="revoke-keys"
Oct 30 17:42:05.474821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.474760    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2de424-b8da-44c3-a954-5ca9e1545984" containerName="revoke-keys"
Oct 30 17:42:05.474959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.474946    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2de424-b8da-44c3-a954-5ca9e1545984" containerName="revoke-keys"
Oct 30 17:42:05.479465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.479442    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.504512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.504483    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-74762d-serving-cert\""
Oct 30 17:42:05.506807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.506770    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"]
Oct 30 17:42:05.521181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.521135    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.521369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.521348    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.521556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.521535    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.521668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.521580    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxpb\" (UniqueName: \"kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.557121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.557089    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:05.623126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.623088    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.623335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.623176    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.623335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.623202    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxpb\" (UniqueName: \"kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.623335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.623308    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.624264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.624232    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.625946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.625921    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.626318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.626152    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.647270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.647233    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxpb\" (UniqueName: \"kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb\") pod \"maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.670721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.670680    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:05.789041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.788943    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:05.814705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.812511    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:05.963383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:05.963351    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:06.149543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:06.149496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:06.150742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:06.150718    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"]
Oct 30 17:42:06.150903 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:42:06.150882    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0c353c_77b2_4b87_a22c_9e723a6de0ca.slice/crio-992ed019983794a9a4bc712ba1bd40c4c971c748f0d3dadf4cae01e062bde799 WatchSource:0}: Error finding container 992ed019983794a9a4bc712ba1bd40c4c971c748f0d3dadf4cae01e062bde799: Status 404 returned error can't find the container with id 992ed019983794a9a4bc712ba1bd40c4c971c748f0d3dadf4cae01e062bde799
Oct 30 17:42:06.418578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:06.418393    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:06.525673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:06.522541    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw" event={"ID":"6d0c353c-77b2-4b87-a22c-9e723a6de0ca","Type":"ContainerStarted","Data":"992ed019983794a9a4bc712ba1bd40c4c971c748f0d3dadf4cae01e062bde799"}
Oct 30 17:42:06.577561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:06.577520    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:06.919285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:06.919245    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:07.380178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:07.380144    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:07.536017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:07.535924    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw" event={"ID":"6d0c353c-77b2-4b87-a22c-9e723a6de0ca","Type":"ContainerStarted","Data":"73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834"}
Oct 30 17:42:07.536620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:07.536462    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:07.585665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:07.584911    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw" podStartSLOduration=2.088147972 podStartE2EDuration="2.584889536s" totalImagesPullingTime="496.741564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:07.57829376 +0000 UTC m=+1733.340337577" watchObservedRunningTime="2026-10-30 17:42:07.584889536 +0000 UTC m=+1733.346933350"
Oct 30 17:42:07.644059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:07.643627    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:07.766576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:07.766538    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:09.492018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:09.491983    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:09.613811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:09.613765    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:09.726369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:09.726329    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:10.223929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:10.223885    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"]
Oct 30 17:42:10.224206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:10.224172    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy" containerID="cri-o://c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a" gracePeriod=30
Oct 30 17:42:10.503566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:10.503418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:10.639095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:10.639058    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:10.993663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:10.993590    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:11.164129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:11.164047    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:11.305516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:11.305416    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:11.826268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:11.826236    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:12.828709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.828673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:12.950469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.950437    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"]
Oct 30 17:42:12.952247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.952211    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:12.955479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.955458    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:12.986009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.985959    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-874b83\""
Oct 30 17:42:12.986009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.985989    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-874b83-dockercfg-bdzdf\""
Oct 30 17:42:12.990438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:12.990405    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"]
Oct 30 17:42:13.118059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.117752    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"]
Oct 30 17:42:13.123174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.123138    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxzlt\" (UniqueName: \"kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt\") pod \"payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.123377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.123218    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.134322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.134294    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"]
Oct 30 17:42:13.134673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.134630    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.227776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.224961    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-874b83-56655fc968-p55vn\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.227776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.225123    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscpw\" (UniqueName: \"kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw\") pod \"payload-processing-e2e-inf-a-874b83-56655fc968-p55vn\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.227776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.225170    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxzlt\" (UniqueName: \"kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt\") pod \"payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.227776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.225215    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.227776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.226023    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.281387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.281185    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxzlt\" (UniqueName: \"kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt\") pod \"payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.326475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.326439    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscpw\" (UniqueName: \"kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw\") pod \"payload-processing-e2e-inf-a-874b83-56655fc968-p55vn\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.326683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.326663    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-874b83-56655fc968-p55vn\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.327415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.327392    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-874b83-56655fc968-p55vn\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.353101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.353032    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscpw\" (UniqueName: \"kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw\") pod \"payload-processing-e2e-inf-a-874b83-56655fc968-p55vn\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.480860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.480767    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:13.552811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.552785    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:42:13.573264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.573228    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:13.747589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.747504    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:13.829322 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:42:13.829187    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67499af8_ea4a_4461_a5f8_e58d0354893f.slice/crio-adbacfc40d44c7cef87c2edf38f0fd71aad8ee7e5c1e414bb3a9fb305246a3c2 WatchSource:0}: Error finding container adbacfc40d44c7cef87c2edf38f0fd71aad8ee7e5c1e414bb3a9fb305246a3c2: Status 404 returned error can't find the container with id adbacfc40d44c7cef87c2edf38f0fd71aad8ee7e5c1e414bb3a9fb305246a3c2
Oct 30 17:42:13.832484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.832456    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"]
Oct 30 17:42:13.921345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:13.921268    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"]
Oct 30 17:42:13.921452 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:42:13.921312    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba263fbd_edcd_4193_9baf_f0629b2bae42.slice/crio-3ff9ff20491082e05f4caa17395ef87431dff579db5f0851724951c5110eac8c WatchSource:0}: Error finding container 3ff9ff20491082e05f4caa17395ef87431dff579db5f0851724951c5110eac8c: Status 404 returned error can't find the container with id 3ff9ff20491082e05f4caa17395ef87431dff579db5f0851724951c5110eac8c
Oct 30 17:42:14.506215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.506180    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:14.612589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.612558    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"]
Oct 30 17:42:14.617303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.617284    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.619396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.619375    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:14.630476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.630444    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn" event={"ID":"67499af8-ea4a-4461-a5f8-e58d0354893f","Type":"ContainerStarted","Data":"0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76"}
Oct 30 17:42:14.630594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.630497    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn" event={"ID":"67499af8-ea4a-4461-a5f8-e58d0354893f","Type":"ContainerStarted","Data":"adbacfc40d44c7cef87c2edf38f0fd71aad8ee7e5c1e414bb3a9fb305246a3c2"}
Oct 30 17:42:14.630753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.630734    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:14.634619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.634587    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" event={"ID":"ba263fbd-edcd-4193-9baf-f0629b2bae42","Type":"ContainerStarted","Data":"abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06"}
Oct 30 17:42:14.634774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.634695    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" event={"ID":"ba263fbd-edcd-4193-9baf-f0629b2bae42","Type":"ContainerStarted","Data":"3ff9ff20491082e05f4caa17395ef87431dff579db5f0851724951c5110eac8c"}
Oct 30 17:42:14.636360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.635808    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:14.636360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.635850    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f8b48d\""
Oct 30 17:42:14.638194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.638173    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"]
Oct 30 17:42:14.687668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.686787    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"]
Oct 30 17:42:14.693310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.693283    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:14.710971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.710311    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f8b48d-dockercfg-v8n9t\""
Oct 30 17:42:14.710971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.710606    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"]
Oct 30 17:42:14.711677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.711613    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" podStartSLOduration=2.711600726 podStartE2EDuration="2.711600726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:14.710456474 +0000 UTC m=+1740.472500288" watchObservedRunningTime="2026-10-30 17:42:14.711600726 +0000 UTC m=+1740.473644539"
Oct 30 17:42:14.752543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.752498    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfrkg\" (UniqueName: \"kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg\") pod \"payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.752774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.752570    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.755919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.755858    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn" podStartSLOduration=1.755840874 podStartE2EDuration="1.755840874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:14.750796176 +0000 UTC m=+1740.512839989" watchObservedRunningTime="2026-10-30 17:42:14.755840874 +0000 UTC m=+1740.517884687"
Oct 30 17:42:14.853950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.853913    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:14.854428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.854106    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfrkg\" (UniqueName: \"kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg\") pod \"payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.854428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.854160    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.854428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.854202    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqqq\" (UniqueName: \"kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq\") pod \"payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:14.855055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.855031    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.926501 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.925804    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfrkg\" (UniqueName: \"kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg\") pod \"payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.971430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.971393    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:14.972726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.972692    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqqq\" (UniqueName: \"kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq\") pod \"payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:14.973087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.973060    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:14.973997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:14.973973    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:15.004665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.001805    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqqq\" (UniqueName: \"kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq\") pod \"payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:15.013783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.013744    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:15.261827 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:42:15.261789    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:42:15.400417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.399916    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"]
Oct 30 17:42:15.417972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.416519    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"]
Oct 30 17:42:15.509316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.508993    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:15.540584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.540558    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:42:15.687975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.687879    5534 generic.go:356] "Generic (PLEG): container finished" podID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerID="c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a" exitCode=0
Oct 30 17:42:15.688292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.688263    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" event={"ID":"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873","Type":"ContainerDied","Data":"c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a"}
Oct 30 17:42:15.688541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.688518    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd" event={"ID":"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873","Type":"ContainerDied","Data":"61e7b1d4b88985e58e81890b1ad39fd8acb06268ca299ef0ff20f6ac0f1841f2"}
Oct 30 17:42:15.688740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.688722    5534 scope.go:122] "RemoveContainer" containerID="c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a"
Oct 30 17:42:15.689121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.689100    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"
Oct 30 17:42:15.692793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.692765    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" event={"ID":"9660209e-2cbf-4834-824b-005c96f6e624","Type":"ContainerStarted","Data":"f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128"}
Oct 30 17:42:15.692906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.692838    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" event={"ID":"9660209e-2cbf-4834-824b-005c96f6e624","Type":"ContainerStarted","Data":"24a60a0eac081afb86be0d803085fa8d7b9070a0e8cf36690819a6a89ae75307"}
Oct 30 17:42:15.693758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.693736    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:15.710664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.707698    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" event={"ID":"188e41bd-2374-4dc9-9ba5-824ada2f7647","Type":"ContainerStarted","Data":"e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68"}
Oct 30 17:42:15.710664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.707779    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" event={"ID":"188e41bd-2374-4dc9-9ba5-824ada2f7647","Type":"ContainerStarted","Data":"1f5a7b4e8c92b49589a08a8cefedec158d7453257b99daf7b83a5c07c4805604"}
Oct 30 17:42:15.710664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.709054    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:15.716259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.715992    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.716259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.716056    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.716259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.716126    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.716259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.716161    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.716624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.716320    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:42:15.716624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.716511    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.716941    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.717050    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.717102    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.717231    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.717300    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket\") pod \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\" (UID: \"3e9ce9aa-bb0f-494e-805a-eaa70fc5b873\") "
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.718096    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.718348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.718120    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.718930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.718534    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:42:15.718930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.718823    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:42:15.719214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.719168    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:42:15.719421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.719273    5534 scope.go:122] "RemoveContainer" containerID="c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a"
Oct 30 17:42:15.719869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.719512    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "kube-api-access-4nkcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:42:15.719869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.719556    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:42:15.720279 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:42:15.720191    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a\": container with ID starting with c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a not found: ID does not exist" containerID="c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a"
Oct 30 17:42:15.720279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.720234    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a"} err="failed to get container status \"c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a\": rpc error: code = NotFound desc = could not find container \"c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a\": container with ID starting with c6023c1fe18b8986273e051f6b5bf41246e983fc0e2b4e01d19a4c55d816993a not found: ID does not exist"
Oct 30 17:42:15.720485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.720463    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:42:15.721560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.721536    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy" pod "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" (UID: "3e9ce9aa-bb0f-494e-805a-eaa70fc5b873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:42:15.756699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.756614    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" podStartSLOduration=1.7565981320000001 podStartE2EDuration="1.756598132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:15.753318227 +0000 UTC m=+1741.515362040" watchObservedRunningTime="2026-10-30 17:42:15.756598132 +0000 UTC m=+1741.518641945"
Oct 30 17:42:15.797617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.797542    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" podStartSLOduration=1.797521422 podStartE2EDuration="1.797521422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:15.79303324 +0000 UTC m=+1741.555077053" watchObservedRunningTime="2026-10-30 17:42:15.797521422 +0000 UTC m=+1741.559565237"
Oct 30 17:42:15.820355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820324    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.820355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820351    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.820554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820381    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.820554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820397    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.820554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820405    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.820554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820415    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nkcb\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-kube-api-access-4nkcb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:15.820554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:15.820438    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:16.043691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:16.043611    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"]
Oct 30 17:42:16.049446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:16.049412    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7lsvd"]
Oct 30 17:42:16.374373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:16.374332    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" path="/var/lib/kubelet/pods/3e9ce9aa-bb0f-494e-805a-eaa70fc5b873/volumes"
Oct 30 17:42:16.794152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:16.794113    5534 scope.go:122] "RemoveContainer" containerID="f2fded00bcf3fd74c52edab7368c1bd22c9f685a4e8f3c5075530d519947b5b2"
Oct 30 17:42:16.806069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:16.806048    5534 scope.go:122] "RemoveContainer" containerID="3dfc11cd0c154f3dcf434e18ce1cedc2ba0f97ca1b643c71469b29909305b4c6"
Oct 30 17:42:17.406471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:17.406441    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:19.202763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:19.202728    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:20.896494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:20.896460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:22.267824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:22.267789    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:22.706922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:22.706887    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:24.292851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.292817    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"]
Oct 30 17:42:24.293604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.293590    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy"
Oct 30 17:42:24.293708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.293605    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy"
Oct 30 17:42:24.293756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.293722    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9ce9aa-bb0f-494e-805a-eaa70fc5b873" containerName="istio-proxy"
Oct 30 17:42:24.299500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.299463    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.328468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.328438    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"]
Oct 30 17:42:24.331703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.331634    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-38f445-serving-cert\""
Oct 30 17:42:24.351397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.351365    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.351699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.351678    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.352022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.352006    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.352258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.352244    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcf8\" (UniqueName: \"kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.424049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.424013    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:24.453484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.453444    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.453707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.453615    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcf8\" (UniqueName: \"kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.453780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.453756    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.453845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.453806    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.454545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.454519    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.456247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.456223    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.456824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.456803    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.481157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.481124    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcf8\" (UniqueName: \"kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8\") pod \"maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.575732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.573021    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:24.614416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.614384    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:24.727210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.725068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:24.962352 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:42:24.962319    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac41727_4af1_48f9_b41d_2b7186f34a6d.slice/crio-9e12ece60bf060bac0d712403f9a6dac5183c834d37a3d1b8d52afe9a76941ba WatchSource:0}: Error finding container 9e12ece60bf060bac0d712403f9a6dac5183c834d37a3d1b8d52afe9a76941ba: Status 404 returned error can't find the container with id 9e12ece60bf060bac0d712403f9a6dac5183c834d37a3d1b8d52afe9a76941ba
Oct 30 17:42:24.962507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:24.962427    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"]
Oct 30 17:42:25.714846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:25.714811    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:42:25.819752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:25.817664    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" event={"ID":"7ac41727-4af1-48f9-b41d-2b7186f34a6d","Type":"ContainerStarted","Data":"b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"}
Oct 30 17:42:25.819752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:25.817747    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" event={"ID":"7ac41727-4af1-48f9-b41d-2b7186f34a6d","Type":"ContainerStarted","Data":"9e12ece60bf060bac0d712403f9a6dac5183c834d37a3d1b8d52afe9a76941ba"}
Oct 30 17:42:25.819752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:25.818932    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:25.876902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:25.876849    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" podStartSLOduration=1.568528858 podStartE2EDuration="1.876834009s" totalImagesPullingTime="308.305151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:25.873972756 +0000 UTC m=+1751.636016568" watchObservedRunningTime="2026-10-30 17:42:25.876834009 +0000 UTC m=+1751.638877821"
Oct 30 17:42:26.334185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:26.334149    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:26.723507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:26.723474    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:42:26.724018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:26.723564    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:42:27.731961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:27.731931    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:42:28.155003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:28.154951    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:29.953396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:29.953362    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:30.908348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:30.908313    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:31.570013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:31.569982    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:32.854556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:32.854512    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.181:8443/health\": dial tcp 10.133.0.181:8443: connect: connection refused"
Oct 30 17:42:33.431997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:33.431961    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:33.582590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:33.582549    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:34.364154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:34.364117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:35.077349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:35.077318    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:35.933578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:35.933540    5534 generic.go:356] "Generic (PLEG): container finished" podID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerID="b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1" exitCode=1
Oct 30 17:42:35.934047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:35.933669    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" event={"ID":"7ac41727-4af1-48f9-b41d-2b7186f34a6d","Type":"ContainerDied","Data":"b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"}
Oct 30 17:42:35.934201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:35.934180    5534 scope.go:122] "RemoveContainer" containerID="b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"
Oct 30 17:42:36.754604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:36.754572    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:36.943574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:36.943535    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" event={"ID":"7ac41727-4af1-48f9-b41d-2b7186f34a6d","Type":"ContainerStarted","Data":"99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96"}
Oct 30 17:42:36.944105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:36.943789    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:38.061856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.061823    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"]
Oct 30 17:42:38.063083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.063048    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j" podUID="f849c571-d2c8-457f-a01b-ebbc08bbe325" containerName="maas-api" containerID="cri-o://4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8" gracePeriod=30
Oct 30 17:42:38.114902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.114869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:38.426174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.426142    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:42:38.530672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.530252    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls\") pod \"f849c571-d2c8-457f-a01b-ebbc08bbe325\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") "
Oct 30 17:42:38.530672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.530325    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs\") pod \"f849c571-d2c8-457f-a01b-ebbc08bbe325\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") "
Oct 30 17:42:38.530672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.530440    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2\") pod \"f849c571-d2c8-457f-a01b-ebbc08bbe325\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") "
Oct 30 17:42:38.530672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.530484    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca\") pod \"f849c571-d2c8-457f-a01b-ebbc08bbe325\" (UID: \"f849c571-d2c8-457f-a01b-ebbc08bbe325\") "
Oct 30 17:42:38.543118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.539032    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca" pod "f849c571-d2c8-457f-a01b-ebbc08bbe325" (UID: "f849c571-d2c8-457f-a01b-ebbc08bbe325"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:42:38.561531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.557550    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f849c571-d2c8-457f-a01b-ebbc08bbe325-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:38.561531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.561094    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs" pod "f849c571-d2c8-457f-a01b-ebbc08bbe325" (UID: "f849c571-d2c8-457f-a01b-ebbc08bbe325"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:42:38.578460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.562161    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls" pod "f849c571-d2c8-457f-a01b-ebbc08bbe325" (UID: "f849c571-d2c8-457f-a01b-ebbc08bbe325"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:42:38.584884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.584827    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2" pod "f849c571-d2c8-457f-a01b-ebbc08bbe325" (UID: "f849c571-d2c8-457f-a01b-ebbc08bbe325"). InnerVolumeSpecName "kube-api-access-6rzv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:42:38.689894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.689072    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:38.689894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.689140    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/f849c571-d2c8-457f-a01b-ebbc08bbe325-kube-api-access-6rzv2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:38.689894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.689158    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f849c571-d2c8-457f-a01b-ebbc08bbe325-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:38.699159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.695793    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:38.963799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.963710    5534 generic.go:356] "Generic (PLEG): container finished" podID="f849c571-d2c8-457f-a01b-ebbc08bbe325" containerID="4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8" exitCode=0
Oct 30 17:42:38.963799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.963779    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"
Oct 30 17:42:38.964006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.963796    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j" event={"ID":"f849c571-d2c8-457f-a01b-ebbc08bbe325","Type":"ContainerDied","Data":"4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8"}
Oct 30 17:42:38.964006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.963880    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j" event={"ID":"f849c571-d2c8-457f-a01b-ebbc08bbe325","Type":"ContainerDied","Data":"de6381da4fe57acb4300274fddf553de2c7b214a671b3ca18955ff6c6fbce3d3"}
Oct 30 17:42:38.964006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.963916    5534 scope.go:122] "RemoveContainer" containerID="4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8"
Oct 30 17:42:38.976629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.976608    5534 scope.go:122] "RemoveContainer" containerID="4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8"
Oct 30 17:42:38.981261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.977160    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:38.981261 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:42:38.979779    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8\": container with ID starting with 4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8 not found: ID does not exist" containerID="4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8"
Oct 30 17:42:38.981261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:38.979815    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8"} err="failed to get container status \"4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8\": rpc error: code = NotFound desc = could not find container \"4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8\": container with ID starting with 4dd35a63a16f4b94c5a67ad15b71730c6113aaddef950b10e15990dca53413a8 not found: ID does not exist"
Oct 30 17:42:39.032567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.032536    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"]
Oct 30 17:42:39.036186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.036154    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"]
Oct 30 17:42:39.040105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.040080    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9bdd9-57cb6585df-dx54j"]
Oct 30 17:42:39.066664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.057131    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9bdd9-snbtj"]
Oct 30 17:42:39.091544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.091511    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:39.202996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.202965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:39.559243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:39.559211    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:40.373795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:40.373761    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57ab26d-0055-4a0e-972d-e11c0a92839f" path="/var/lib/kubelet/pods/f57ab26d-0055-4a0e-972d-e11c0a92839f/volumes"
Oct 30 17:42:40.374817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:40.374794    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f849c571-d2c8-457f-a01b-ebbc08bbe325" path="/var/lib/kubelet/pods/f849c571-d2c8-457f-a01b-ebbc08bbe325/volumes"
Oct 30 17:42:40.997654 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:40.997615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:42.720626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:42.720520    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:42.985736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:42.983856    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:42:44.823179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:44.823135    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:45.249029 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:42:45.248992    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:42:45.837033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:45.837004    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:45.973686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:45.973626    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:46.248724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.248697    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:46.462223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.462039    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:46.837590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.837550    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"]
Oct 30 17:42:46.842194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.839390    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f849c571-d2c8-457f-a01b-ebbc08bbe325" containerName="maas-api"
Oct 30 17:42:46.842194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.839417    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f849c571-d2c8-457f-a01b-ebbc08bbe325" containerName="maas-api"
Oct 30 17:42:46.842194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.839676    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f849c571-d2c8-457f-a01b-ebbc08bbe325" containerName="maas-api"
Oct 30 17:42:46.849209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.849180    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"]
Oct 30 17:42:46.849526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.849506    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:46.871103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.869146    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:42:46.901758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.891024    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:42:46.938713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.938622    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hwq\" (UniqueName: \"kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:46.938924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.938790    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:46.939040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:46.939003    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.040662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.040596    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hwq\" (UniqueName: \"kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.040825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.040698    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.040825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.040818    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.041555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.041531    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.047683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.043706    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.110463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.110424    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hwq\" (UniqueName: \"kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq\") pod \"maas-api-revoke-keys-e2e-api-b-49e155-hdpc5\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.237773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.237733    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:47.628348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:47.628299    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:48.042333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.042087    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"]
Oct 30 17:42:48.046130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.046097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:48.208878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.208295    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5" event={"ID":"61184f05-aa07-444a-bc0e-fd6a0f278fdd","Type":"ContainerStarted","Data":"094cb19f589af0f5524e74006bcf5f4358ee3690cfc6959d381461de0c213346"}
Oct 30 17:42:48.618679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.618597    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"]
Oct 30 17:42:48.625731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.624788    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.629740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.628885    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"]
Oct 30 17:42:48.751906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.751864    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkhs\" (UniqueName: \"kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.752107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.752083    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.752191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.752147    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.864816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.864769    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkhs\" (UniqueName: \"kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.865021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.864986    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.865092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.865065    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.866728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.866698    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.868412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.868386    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.906608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.901546    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkhs\" (UniqueName: \"kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs\") pod \"maas-api-revoke-keys-e2e-api-a-466900-pgpjm\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:48.945626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:48.945346    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:49.222547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:49.222463    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5" event={"ID":"61184f05-aa07-444a-bc0e-fd6a0f278fdd","Type":"ContainerStarted","Data":"f8aedcdba41bd9cfaf6c6eed33bcd884bf07865a1bf3c0a32db986eb36d06583"}
Oct 30 17:42:49.282829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:49.282774    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5" podStartSLOduration=3.282755919 podStartE2EDuration="3.282755919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:49.279512258 +0000 UTC m=+1775.041556094" watchObservedRunningTime="2026-10-30 17:42:49.282755919 +0000 UTC m=+1775.044799734"
Oct 30 17:42:49.288312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:49.287836    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"]
Oct 30 17:42:49.293106 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:42:49.293066    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309fff25_a695_4968_8987_bd2f8166d0a1.slice/crio-e332b6b1c90d04b79742366208d62482b36d1662426f7728f003dd6b13d534df WatchSource:0}: Error finding container e332b6b1c90d04b79742366208d62482b36d1662426f7728f003dd6b13d534df: Status 404 returned error can't find the container with id e332b6b1c90d04b79742366208d62482b36d1662426f7728f003dd6b13d534df
Oct 30 17:42:50.103730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:50.103597    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:50.234502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:50.234462    5534 generic.go:356] "Generic (PLEG): container finished" podID="61184f05-aa07-444a-bc0e-fd6a0f278fdd" containerID="f8aedcdba41bd9cfaf6c6eed33bcd884bf07865a1bf3c0a32db986eb36d06583" exitCode=0
Oct 30 17:42:50.235046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:50.234556    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5" event={"ID":"61184f05-aa07-444a-bc0e-fd6a0f278fdd","Type":"ContainerDied","Data":"f8aedcdba41bd9cfaf6c6eed33bcd884bf07865a1bf3c0a32db986eb36d06583"}
Oct 30 17:42:50.236937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:50.236909    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm" event={"ID":"309fff25-a695-4968-8987-bd2f8166d0a1","Type":"ContainerStarted","Data":"622878a54d6d239213873ce6aff96aaee6af8c8599459c06fbc50709821c3acf"}
Oct 30 17:42:50.237072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:50.236971    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm" event={"ID":"309fff25-a695-4968-8987-bd2f8166d0a1","Type":"ContainerStarted","Data":"e332b6b1c90d04b79742366208d62482b36d1662426f7728f003dd6b13d534df"}
Oct 30 17:42:50.323599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:50.323545    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm" podStartSLOduration=2.323524688 podStartE2EDuration="2.323524688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:50.318101041 +0000 UTC m=+1776.080144875" watchObservedRunningTime="2026-10-30 17:42:50.323524688 +0000 UTC m=+1776.085568502"
Oct 30 17:42:51.246767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.246727    5534 generic.go:356] "Generic (PLEG): container finished" podID="309fff25-a695-4968-8987-bd2f8166d0a1" containerID="622878a54d6d239213873ce6aff96aaee6af8c8599459c06fbc50709821c3acf" exitCode=0
Oct 30 17:42:51.247187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.246817    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm" event={"ID":"309fff25-a695-4968-8987-bd2f8166d0a1","Type":"ContainerDied","Data":"622878a54d6d239213873ce6aff96aaee6af8c8599459c06fbc50709821c3acf"}
Oct 30 17:42:51.572014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.571930    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:51.703744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.699405    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca\") pod \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") "
Oct 30 17:42:51.703744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.699516    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca\") pod \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") "
Oct 30 17:42:51.703744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.699767    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq\" (UniqueName: \"kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq\") pod \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\" (UID: \"61184f05-aa07-444a-bc0e-fd6a0f278fdd\") "
Oct 30 17:42:51.739672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.738249    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca" pod "61184f05-aa07-444a-bc0e-fd6a0f278fdd" (UID: "61184f05-aa07-444a-bc0e-fd6a0f278fdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:42:51.739672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.739171    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:51.880584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.878408    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq" pod "61184f05-aa07-444a-bc0e-fd6a0f278fdd" (UID: "61184f05-aa07-444a-bc0e-fd6a0f278fdd"). InnerVolumeSpecName "kube-api-access-p7hwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:42:51.885584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.885549    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca" pod "61184f05-aa07-444a-bc0e-fd6a0f278fdd" (UID: "61184f05-aa07-444a-bc0e-fd6a0f278fdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:42:51.943764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.943731    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hwq\" (UniqueName: \"kubernetes.io/projected/61184f05-aa07-444a-bc0e-fd6a0f278fdd-kube-api-access-p7hwq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:51.944012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:51.943998    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61184f05-aa07-444a-bc0e-fd6a0f278fdd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:52.093162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.093114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:52.227665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.226909    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:52.266016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.263134    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"
Oct 30 17:42:52.266016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.263531    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5" event={"ID":"61184f05-aa07-444a-bc0e-fd6a0f278fdd","Type":"ContainerDied","Data":"094cb19f589af0f5524e74006bcf5f4358ee3690cfc6959d381461de0c213346"}
Oct 30 17:42:52.266016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.263578    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094cb19f589af0f5524e74006bcf5f4358ee3690cfc6959d381461de0c213346"
Oct 30 17:42:52.428329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.428294    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"]
Oct 30 17:42:52.429541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.429513    5534 cpu_manager.go:414] "containerMap: removing container" podUID="61184f05-aa07-444a-bc0e-fd6a0f278fdd" containerName="revoke-keys"
Oct 30 17:42:52.429541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.429540    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61184f05-aa07-444a-bc0e-fd6a0f278fdd" containerName="revoke-keys"
Oct 30 17:42:52.429826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.429808    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="61184f05-aa07-444a-bc0e-fd6a0f278fdd" containerName="revoke-keys"
Oct 30 17:42:52.452524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.452413    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.467966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.454410    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"]
Oct 30 17:42:52.548299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.548269    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:52.580588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.580545    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.580778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.580749    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.580919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.580898    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.580960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.580943    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.581000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.580976    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmxhm\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.581045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.581030    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.581097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.581083    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.581132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.581122    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.581170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.581158    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.682405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.682364    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca\") pod \"309fff25-a695-4968-8987-bd2f8166d0a1\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") "
Oct 30 17:42:52.682621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.682447    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca\") pod \"309fff25-a695-4968-8987-bd2f8166d0a1\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") "
Oct 30 17:42:52.682723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.682618    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs\" (UniqueName: \"kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs\") pod \"309fff25-a695-4968-8987-bd2f8166d0a1\" (UID: \"309fff25-a695-4968-8987-bd2f8166d0a1\") "
Oct 30 17:42:52.682991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.682964    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683017    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683052    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmxhm\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683153    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683218    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683258    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683298    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683368    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.683524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.683508    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.684219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.684193    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.686020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.685989    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca" pod "309fff25-a695-4968-8987-bd2f8166d0a1" (UID: "309fff25-a695-4968-8987-bd2f8166d0a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:42:52.686129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.686041    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.686349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.686326    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.686476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.686430    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.688622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.686683    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.690559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.689303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.690778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.690731    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca" pod "309fff25-a695-4968-8987-bd2f8166d0a1" (UID: "309fff25-a695-4968-8987-bd2f8166d0a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:42:52.692693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.691348    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs" pod "309fff25-a695-4968-8987-bd2f8166d0a1" (UID: "309fff25-a695-4968-8987-bd2f8166d0a1"). InnerVolumeSpecName "kube-api-access-lrkhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:42:52.692693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.692248    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.712116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.712087    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmxhm\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.712277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.712115    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:52.785574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.785483    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:52.785574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.785523    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309fff25-a695-4968-8987-bd2f8166d0a1-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:52.785574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.785533    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrkhs\" (UniqueName: \"kubernetes.io/projected/309fff25-a695-4968-8987-bd2f8166d0a1-kube-api-access-lrkhs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:42:52.814783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:52.814744    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:53.202125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.200567    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"]
Oct 30 17:42:53.206849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.206805    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:53.206990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.206899    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:53.206990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.206942    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:53.279184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.277792    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm" event={"ID":"309fff25-a695-4968-8987-bd2f8166d0a1","Type":"ContainerDied","Data":"e332b6b1c90d04b79742366208d62482b36d1662426f7728f003dd6b13d534df"}
Oct 30 17:42:53.279184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.278204    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e332b6b1c90d04b79742366208d62482b36d1662426f7728f003dd6b13d534df"
Oct 30 17:42:53.279184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.278042    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"
Oct 30 17:42:53.287138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.287052    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" event={"ID":"4c9570be-a0d3-4b42-849d-9523b3f794f9","Type":"ContainerStarted","Data":"9e55b707a29b916fc57c86e769185ed7fc150a79b985084ec7f2398b4df5193d"}
Oct 30 17:42:53.314965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.314925    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:53.832429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:53.832397    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:54.235512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.214743    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": read tcp 10.133.0.2:49796->10.133.0.164:15021: read: connection reset by peer" start-of-body=
Oct 30 17:42:54.235512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.214807    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": read tcp 10.133.0.2:49796->10.133.0.164:15021: read: connection reset by peer"
Oct 30 17:42:54.301681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.301620    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"]
Oct 30 17:42:54.304848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.303836    5534 cpu_manager.go:414] "containerMap: removing container" podUID="309fff25-a695-4968-8987-bd2f8166d0a1" containerName="revoke-keys"
Oct 30 17:42:54.304848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.303860    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309fff25-a695-4968-8987-bd2f8166d0a1" containerName="revoke-keys"
Oct 30 17:42:54.304848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.304592    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="309fff25-a695-4968-8987-bd2f8166d0a1" containerName="revoke-keys"
Oct 30 17:42:54.343360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.343044    5534 generic.go:356] "Generic (PLEG): container finished" podID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerID="74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239" exitCode=0
Oct 30 17:42:54.347154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.347109    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" event={"ID":"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b","Type":"ContainerDied","Data":"74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"}
Oct 30 17:42:54.347154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.347158    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" event={"ID":"4c9570be-a0d3-4b42-849d-9523b3f794f9","Type":"ContainerStarted","Data":"e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0"}
Oct 30 17:42:54.347375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.347186    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"]
Oct 30 17:42:54.347375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.347285    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.347841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.347819    5534 scope.go:122] "RemoveContainer" containerID="74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"
Oct 30 17:42:54.352486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.352449    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:54.352567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.352538    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:54.352622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.352583    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:54.391291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.390839    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-143041-openshift-default-dockercfg-j9lzf\""
Oct 30 17:42:54.391291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.391155    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.543939    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544046    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdhh\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544087    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544240    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544267    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544371    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544515    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544623    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.547664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.544794    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.634702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.607215    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:54.647222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.647162    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.647663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.647607    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.647898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.647852    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.648134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.648091    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.648369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.648293    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdhh\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.648369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.648331    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.649118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.648765    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.649118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.648801    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.649118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.648881    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.650045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.650004    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.650143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.650015    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.651139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.650928    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.651139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.651045    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.651139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.651099    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.690258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.679939    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podStartSLOduration=2.679914276 podStartE2EDuration="2.679914276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:54.609902711 +0000 UTC m=+1780.371946554" watchObservedRunningTime="2026-10-30 17:42:54.679914276 +0000 UTC m=+1780.441958091"
Oct 30 17:42:54.737748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.736888    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.737748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.737084    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.739921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.739856    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.743787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.743718    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdhh\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:54.835526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.835482    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:42:54.836414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.836367    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 17:42:54.836573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:54.836426    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 17:42:55.027993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.027964    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:55.388326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.388228    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" event={"ID":"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b","Type":"ContainerStarted","Data":"560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05"}
Oct 30 17:42:55.390382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.390365    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:42:55.390860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.390831    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:42:55.390967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.390885    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:42:55.391581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.391557    5534 generic.go:356] "Generic (PLEG): container finished" podID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerID="f1baf28d949b9804f2b36ef43e248c529f846650008ca9d53f9037295b3cc6ae" exitCode=0
Oct 30 17:42:55.391696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.391672    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" event={"ID":"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439","Type":"ContainerDied","Data":"f1baf28d949b9804f2b36ef43e248c529f846650008ca9d53f9037295b3cc6ae"}
Oct 30 17:42:55.392121 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.392099    5534 scope.go:122] "RemoveContainer" containerID="f1baf28d949b9804f2b36ef43e248c529f846650008ca9d53f9037295b3cc6ae"
Oct 30 17:42:55.394010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.393980    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:55.394128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.394081    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:55.394198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.394130    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:55.436052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.436026    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"]
Oct 30 17:42:55.439483 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.439454    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:55.439581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.439515    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:55.439581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.439551    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:42:55.817485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.817032    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 17:42:55.817485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.817090    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 17:42:55.929608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:55.929568    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:56.400669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.400056    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:42:56.400669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.400115    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:42:56.437998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.437967    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" event={"ID":"74ccb873-6839-424e-b006-d07d6bc224a9","Type":"ContainerStarted","Data":"1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"}
Oct 30 17:42:56.438162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.438015    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" event={"ID":"74ccb873-6839-424e-b006-d07d6bc224a9","Type":"ContainerStarted","Data":"0dbbef06d787d6c7c5fa402af5367d0c245efcd8f19683b445a27bf6e20ba308"}
Oct 30 17:42:56.441922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.441898    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" event={"ID":"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439","Type":"ContainerStarted","Data":"a05a16fde58f72fbf16e5e37d8240d09f176bbc1b021ca54fd85da89c44d30dc"}
Oct 30 17:42:56.604671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.600095    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podStartSLOduration=2.600071916 podStartE2EDuration="2.600071916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:56.508126674 +0000 UTC m=+1782.270170500" watchObservedRunningTime="2026-10-30 17:42:56.600071916 +0000 UTC m=+1782.362115730"
Oct 30 17:42:56.822729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.815866    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 17:42:56.822729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:56.815925    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 17:42:57.078030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.052213    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:57.100112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.100079    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:42:57.100455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.100429    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:42:57.100672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.100626    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:42:57.125671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.124557    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:57.152695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.150006    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:57.336889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.329725    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"]
Oct 30 17:42:57.443510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.442295    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:42:57.443510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.442363    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:42:57.543665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.543005    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:57.811903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.809886    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:57.881619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.863614    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:57.919074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.917299    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 17:42:57.919074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.917372    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 17:42:57.949684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:57.921457    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:42:58.135763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.115431    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:42:58.152995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.152939    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:42:58.153250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.153221    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:42:58.418753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.409012    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:42:58.418753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.409092    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:42:58.451231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.450956    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:42:58.487667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.474984    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:42:58.536100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.531010    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:42:58.650665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.646667    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:42:58.650665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.648089    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:42:58.670726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.666708    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 17:42:58.709671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.708743    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 17:42:58.714670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.711462    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 17:42:58.816266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.816228    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 17:42:58.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:58.816523    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 17:42:59.106669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.091807    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:42:59.106669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.091869    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:42:59.145784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.145745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:59.309726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.309100    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:42:59.391722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.390946    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:42:59.391722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.391004    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:42:59.817725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.816962    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 17:42:59.817725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:42:59.817021    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 17:43:00.099668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.095113    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:43:00.099668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.095179    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:43:00.247770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.234675    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"]
Oct 30 17:43:00.330798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.318937    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:00.330798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.330720    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"]
Oct 30 17:43:00.452165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.452093    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:43:00.452362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.452342    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:43:00.486664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.481257    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:00.522668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.521713    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:43:00.670687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:00.654607    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:01.216667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.211579    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:43:01.270672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.217904    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:43:01.270672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.247170    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:43:01.270672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.261453    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:43:01.270672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.265035    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:43:01.392670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.392148    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:43:01.392670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.392211    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:43:01.989669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:01.988862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:02.044090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.044056    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"]
Oct 30 17:43:02.047824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.047799    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcef238-m2sfh"]
Oct 30 17:43:02.098337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.098295    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:43:02.098704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.098668    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:43:02.122989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.122334    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:02.255738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.251802    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"]
Oct 30 17:43:02.325665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.324761    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.340677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.337698    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k" podUID="ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" containerName="payload-processing" containerID="cri-o://ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a" gracePeriod=25
Oct 30 17:43:02.352674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.345897    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-143041-dockercfg-9mz9w\""
Oct 30 17:43:02.352674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.346139    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.352674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.346437    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d5g\" (UniqueName: \"kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g\") pod \"payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.352674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.346760    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"]
Oct 30 17:43:02.371669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.368489    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-143041\""
Oct 30 17:43:02.455665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.396617    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:43:02.455665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.448821    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:43:02.455665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.447596    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d5g\" (UniqueName: \"kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g\") pod \"payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.455665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.449302    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.455665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.450882    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.499666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.483436    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d5g\" (UniqueName: \"kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g\") pod \"payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.510590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.508790    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:02.544347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.543263    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740c0b49-5700-4b79-9f71-508227c47790" path="/var/lib/kubelet/pods/740c0b49-5700-4b79-9f71-508227c47790/volumes"
Oct 30 17:43:02.917903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.917856    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"]
Oct 30 17:43:02.925127 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:02.925095    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec1270c_2741_49f4_9e21_9d53138ad2d7.slice/crio-b03c8adfd11ad372ce9d92650218267beb400a2c01c9d6c52b869e1dd20ba2c2 WatchSource:0}: Error finding container b03c8adfd11ad372ce9d92650218267beb400a2c01c9d6c52b869e1dd20ba2c2: Status 404 returned error can't find the container with id b03c8adfd11ad372ce9d92650218267beb400a2c01c9d6c52b869e1dd20ba2c2
Oct 30 17:43:02.982702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:02.959044    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:43:03.017271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.016817    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl\" (UniqueName: \"kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl\") pod \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") "
Oct 30 17:43:03.017572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.017439    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume\") pod \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\" (UID: \"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1\") "
Oct 30 17:43:03.018348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.018319    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume" pod "ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" (UID: "ee3967f1-65d5-472f-a4ec-e05da2fdd9d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:43:03.019137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.019099    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:03.020440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.020172    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl" pod "ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" (UID: "ee3967f1-65d5-472f-a4ec-e05da2fdd9d1"). InnerVolumeSpecName "kube-api-access-zplbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:43:03.091305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.091272    5534 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 17:43:03.091456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.091325    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 17:43:03.120406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.120368    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplbl\" (UniqueName: \"kubernetes.io/projected/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1-kube-api-access-zplbl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:03.148863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.148824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:03.195100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.194709    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" event={"ID":"aec1270c-2741-49f4-9e21-9d53138ad2d7","Type":"ContainerStarted","Data":"56e8d14e1c84c021e346abd2eb00fc10f7708e5d2d25fe3c84fa5c606110c4af"}
Oct 30 17:43:03.195100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.194782    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" event={"ID":"aec1270c-2741-49f4-9e21-9d53138ad2d7","Type":"ContainerStarted","Data":"b03c8adfd11ad372ce9d92650218267beb400a2c01c9d6c52b869e1dd20ba2c2"}
Oct 30 17:43:03.195100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.194831    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:03.197345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.197307    5534 generic.go:356] "Generic (PLEG): container finished" podID="ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" containerID="ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a" exitCode=0
Oct 30 17:43:03.197497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.197369    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"
Oct 30 17:43:03.197580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.197515    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k" event={"ID":"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1","Type":"ContainerDied","Data":"ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a"}
Oct 30 17:43:03.197580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.197562    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k" event={"ID":"ee3967f1-65d5-472f-a4ec-e05da2fdd9d1","Type":"ContainerDied","Data":"c8784ea7c904c67708d364a7919737575d6aceece437e70b4c32ced8daaa4ccd"}
Oct 30 17:43:03.198771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.197604    5534 scope.go:122] "RemoveContainer" containerID="ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a"
Oct 30 17:43:03.217189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.217134    5534 scope.go:122] "RemoveContainer" containerID="ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a"
Oct 30 17:43:03.219032 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:43:03.219000    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a\": container with ID starting with ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a not found: ID does not exist" containerID="ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a"
Oct 30 17:43:03.219236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.219206    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a"} err="failed to get container status \"ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a\": rpc error: code = NotFound desc = could not find container \"ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a\": container with ID starting with ea896f5e7f85a66c8ede68284a98ccaad86d0ea1e0fdc090df6f15ebdb4f934a not found: ID does not exist"
Oct 30 17:43:03.241140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.241081    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" podStartSLOduration=1.241057515 podStartE2EDuration="1.241057515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:03.236438566 +0000 UTC m=+1788.998482380" watchObservedRunningTime="2026-10-30 17:43:03.241057515 +0000 UTC m=+1789.003101349"
Oct 30 17:43:03.280737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.280693    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"]
Oct 30 17:43:03.288206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.288174    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49e155-8594b46674-dlx7k"]
Oct 30 17:43:03.396923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.396895    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:43:03.397413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.397392    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:43:03.398277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:03.398246    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:43:04.096413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:04.096371    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:43:04.097534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:04.097516    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:43:04.374152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:04.374065    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" path="/var/lib/kubelet/pods/ee3967f1-65d5-472f-a4ec-e05da2fdd9d1/volumes"
Oct 30 17:43:04.968785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:04.968749    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:05.236050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.235914    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl" podUID="6e279f3e-ea68-407b-a205-d437cb4f8433" containerName="payload-processing" containerID="cri-o://319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec" gracePeriod=25
Oct 30 17:43:05.397721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.397680    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd" podUID="ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" containerName="payload-pre-processing" containerID="cri-o://605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2" gracePeriod=25
Oct 30 17:43:05.557469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.557165    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:43:05.665387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.662891    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume\") pod \"6e279f3e-ea68-407b-a205-d437cb4f8433\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") "
Oct 30 17:43:05.665387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.663017    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4\" (UniqueName: \"kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4\") pod \"6e279f3e-ea68-407b-a205-d437cb4f8433\" (UID: \"6e279f3e-ea68-407b-a205-d437cb4f8433\") "
Oct 30 17:43:05.665387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.663372    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume" pod "6e279f3e-ea68-407b-a205-d437cb4f8433" (UID: "6e279f3e-ea68-407b-a205-d437cb4f8433"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:43:05.665387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.663775    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e279f3e-ea68-407b-a205-d437cb4f8433-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:05.675888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.666595    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4" pod "6e279f3e-ea68-407b-a205-d437cb4f8433" (UID: "6e279f3e-ea68-407b-a205-d437cb4f8433"). InnerVolumeSpecName "kube-api-access-gwzt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:43:05.696331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.696306    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:43:05.765630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.765526    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume\") pod \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") "
Oct 30 17:43:05.765630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.765588    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq\" (UniqueName: \"kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq\") pod \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\" (UID: \"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc\") "
Oct 30 17:43:05.766224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.766196    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume" pod "ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" (UID: "ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:43:05.766519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.766492    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:05.766614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.766523    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwzt4\" (UniqueName: \"kubernetes.io/projected/6e279f3e-ea68-407b-a205-d437cb4f8433-kube-api-access-gwzt4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:05.768533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.768501    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq" pod "ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" (UID: "ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc"). InnerVolumeSpecName "kube-api-access-n9ctq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:43:05.868031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:05.867989    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ctq\" (UniqueName: \"kubernetes.io/projected/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc-kube-api-access-n9ctq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:06.232274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.232235    5534 generic.go:356] "Generic (PLEG): container finished" podID="6e279f3e-ea68-407b-a205-d437cb4f8433" containerID="319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec" exitCode=0
Oct 30 17:43:06.232469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.232303    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"
Oct 30 17:43:06.232469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.232320    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl" event={"ID":"6e279f3e-ea68-407b-a205-d437cb4f8433","Type":"ContainerDied","Data":"319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec"}
Oct 30 17:43:06.232469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.232382    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl" event={"ID":"6e279f3e-ea68-407b-a205-d437cb4f8433","Type":"ContainerDied","Data":"2362da12e642e19d1778efe552d734bd86498ed98255ac8e554239846d3b60c5"}
Oct 30 17:43:06.232469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.232406    5534 scope.go:122] "RemoveContainer" containerID="319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec"
Oct 30 17:43:06.234530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.234394    5534 generic.go:356] "Generic (PLEG): container finished" podID="ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" containerID="605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2" exitCode=0
Oct 30 17:43:06.234530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.234460    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"
Oct 30 17:43:06.234530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.234519    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd" event={"ID":"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc","Type":"ContainerDied","Data":"605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2"}
Oct 30 17:43:06.234795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.234597    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd" event={"ID":"ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc","Type":"ContainerDied","Data":"e16261e28c2de32f5ea709a2da0edc396b028069b2334a7a319b182b5a163792"}
Oct 30 17:43:06.244179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.243961    5534 scope.go:122] "RemoveContainer" containerID="319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec"
Oct 30 17:43:06.244381 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:43:06.244357    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec\": container with ID starting with 319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec not found: ID does not exist" containerID="319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec"
Oct 30 17:43:06.244452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.244395    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec"} err="failed to get container status \"319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec\": rpc error: code = NotFound desc = could not find container \"319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec\": container with ID starting with 319cf74c189b3be981c05acb656a16cc442f34f058990a04487ac2434e8faeec not found: ID does not exist"
Oct 30 17:43:06.244452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.244421    5534 scope.go:122] "RemoveContainer" containerID="605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2"
Oct 30 17:43:06.270538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.270513    5534 scope.go:122] "RemoveContainer" containerID="605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2"
Oct 30 17:43:06.270929 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:43:06.270908    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2\": container with ID starting with 605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2 not found: ID does not exist" containerID="605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2"
Oct 30 17:43:06.271004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.270940    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2"} err="failed to get container status \"605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2\": rpc error: code = NotFound desc = could not find container \"605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2\": container with ID starting with 605c7b290f00144be0570c9993a9b82910d23876885b627c622604da838697d2 not found: ID does not exist"
Oct 30 17:43:06.305685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.305625    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"]
Oct 30 17:43:06.315417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.315384    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466900-67f4cf75c9-h7hpd"]
Oct 30 17:43:06.342098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.342067    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"]
Oct 30 17:43:06.345711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.345687    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466900-7c8856645b-l4hdl"]
Oct 30 17:43:06.374557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.374527    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e279f3e-ea68-407b-a205-d437cb4f8433" path="/var/lib/kubelet/pods/6e279f3e-ea68-407b-a205-d437cb4f8433/volumes"
Oct 30 17:43:06.375139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.375119    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" path="/var/lib/kubelet/pods/ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc/volumes"
Oct 30 17:43:06.667865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:06.667822    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:08.514217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:08.514181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:10.112399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:10.112369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:11.784417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:11.774615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:12.830153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:12.830114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:13.399224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.399184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:13.813554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.813472    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"]
Oct 30 17:43:13.814533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814507    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" containerName="payload-pre-processing"
Oct 30 17:43:13.814533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814533    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" containerName="payload-pre-processing"
Oct 30 17:43:13.814779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814560    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" containerName="payload-processing"
Oct 30 17:43:13.814779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814568    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" containerName="payload-processing"
Oct 30 17:43:13.814779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814582    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6e279f3e-ea68-407b-a205-d437cb4f8433" containerName="payload-processing"
Oct 30 17:43:13.814779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814590    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e279f3e-ea68-407b-a205-d437cb4f8433" containerName="payload-processing"
Oct 30 17:43:13.814779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814768    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae839e0e-9fca-4ca5-8dbf-9ae0b43b22cc" containerName="payload-pre-processing"
Oct 30 17:43:13.814779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814782    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3967f1-65d5-472f-a4ec-e05da2fdd9d1" containerName="payload-processing"
Oct 30 17:43:13.815088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.814794    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e279f3e-ea68-407b-a205-d437cb4f8433" containerName="payload-processing"
Oct 30 17:43:13.820338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.820317    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.849812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.849781    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"]
Oct 30 17:43:13.879152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.879124    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:13.908917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.908894    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b1c345-openshift-default-dockercfg-qf5jf\""
Oct 30 17:43:13.915384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915361    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915516    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915557    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnl6\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915578    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915731    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915762    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915790    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915840    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:13.915949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:13.915934    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.009177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.009143    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.017579    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.017663    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.017694    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.017783    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.017854    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.018003    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.018033    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnl6\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.018063    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.018201    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.018685    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.019326    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.042101    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.043138    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.043392    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.059674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.048545    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.077713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.076618    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.111668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.111325    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.116314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.115303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnl6\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6\") pod \"e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") " pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.148668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.142871    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:14.148668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.145635    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:14.221719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.220935    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:43:14.541374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.541346    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"]
Oct 30 17:43:14.542531 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:14.542508    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105ab0a0_4585_4c94_8eb6_c789cd5b00a1.slice/crio-5793fbec0b679b603447765bfaa8100a09de668c0b2c0d0adf1ee2c6b3237130 WatchSource:0}: Error finding container 5793fbec0b679b603447765bfaa8100a09de668c0b2c0d0adf1ee2c6b3237130: Status 404 returned error can't find the container with id 5793fbec0b679b603447765bfaa8100a09de668c0b2c0d0adf1ee2c6b3237130
Oct 30 17:43:14.546614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.546563    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:14.546912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.546881    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:14.547052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:14.546981    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:15.247163 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:43:15.247122    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:43:15.249709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:15.249683    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:15.326696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:15.326660    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" event={"ID":"105ab0a0-4585-4c94-8eb6-c789cd5b00a1","Type":"ContainerStarted","Data":"f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"}
Oct 30 17:43:15.326892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:15.326742    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" event={"ID":"105ab0a0-4585-4c94-8eb6-c789cd5b00a1","Type":"ContainerStarted","Data":"5793fbec0b679b603447765bfaa8100a09de668c0b2c0d0adf1ee2c6b3237130"}
Oct 30 17:43:15.400279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:15.400204    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podStartSLOduration=2.400182795 podStartE2EDuration="2.400182795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:15.384939265 +0000 UTC m=+1801.146983081" watchObservedRunningTime="2026-10-30 17:43:15.400182795 +0000 UTC m=+1801.162226609"
Oct 30 17:43:16.143948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:16.143914    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:16.150212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:16.150158    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:16.167818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:16.167791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:16.285051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:16.284992    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:16.335878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:16.335843    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:16.336860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:16.336839    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:43:17.825793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:17.825763    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:17.985906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:17.985870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:18.157338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:18.157304    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:19.771513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:19.771472    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:19.978195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:19.978162    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"]
Oct 30 17:43:19.984474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:19.984448    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.015951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.015806    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:20.016600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.016575    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b1c345\""
Oct 30 17:43:20.016982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.016966    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b1c345-dockercfg-zwqqj\""
Oct 30 17:43:20.025388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.024167    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"]
Oct 30 17:43:20.028195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.028163    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8ft\" (UniqueName: \"kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft\") pod \"payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.028429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.028411    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.134140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.132624    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.134140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.133034    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8ft\" (UniqueName: \"kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft\") pod \"payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.136590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.136537    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.166461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.166427    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8ft\" (UniqueName: \"kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft\") pod \"payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.211491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.211454    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:20.586709 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:20.586678    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02c7722_8f05_4f8c_ad9b_9c6f1ed63878.slice/crio-cf8c251fe9dfd782a05a0d0b1aa34eeabfb9126359d6c52e51c1a3defe5fe65c WatchSource:0}: Error finding container cf8c251fe9dfd782a05a0d0b1aa34eeabfb9126359d6c52e51c1a3defe5fe65c: Status 404 returned error can't find the container with id cf8c251fe9dfd782a05a0d0b1aa34eeabfb9126359d6c52e51c1a3defe5fe65c
Oct 30 17:43:20.587343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.587319    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"]
Oct 30 17:43:20.748278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:20.748243    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:21.402893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:21.402856    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" event={"ID":"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878","Type":"ContainerStarted","Data":"3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9"}
Oct 30 17:43:21.403311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:21.402916    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" event={"ID":"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878","Type":"ContainerStarted","Data":"cf8c251fe9dfd782a05a0d0b1aa34eeabfb9126359d6c52e51c1a3defe5fe65c"}
Oct 30 17:43:21.403311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:21.402963    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:21.444267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:21.444213    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" podStartSLOduration=2.444198271 podStartE2EDuration="2.444198271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:21.442632535 +0000 UTC m=+1807.204676345" watchObservedRunningTime="2026-10-30 17:43:21.444198271 +0000 UTC m=+1807.206242116"
Oct 30 17:43:22.701522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:22.701480    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:24.383340 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:24.383301    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:25.869896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:25.869856    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"]
Oct 30 17:43:25.870490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:25.870077    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq" podUID="6d881eb0-5569-439f-859c-a86297985e58" containerName="maas-api" containerID="cri-o://9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6" gracePeriod=30
Oct 30 17:43:26.003018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.002991    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:26.247705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.245762    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:43:26.388664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.385195    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p\" (UniqueName: \"kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p\") pod \"6d881eb0-5569-439f-859c-a86297985e58\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") "
Oct 30 17:43:26.388664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.385251    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs\") pod \"6d881eb0-5569-439f-859c-a86297985e58\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") "
Oct 30 17:43:26.388664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.385444    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca\") pod \"6d881eb0-5569-439f-859c-a86297985e58\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") "
Oct 30 17:43:26.388664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.385520    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls\") pod \"6d881eb0-5569-439f-859c-a86297985e58\" (UID: \"6d881eb0-5569-439f-859c-a86297985e58\") "
Oct 30 17:43:26.399974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.399825    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca" pod "6d881eb0-5569-439f-859c-a86297985e58" (UID: "6d881eb0-5569-439f-859c-a86297985e58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:43:26.407088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.407049    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls" pod "6d881eb0-5569-439f-859c-a86297985e58" (UID: "6d881eb0-5569-439f-859c-a86297985e58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:43:26.408794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.408757    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p" pod "6d881eb0-5569-439f-859c-a86297985e58" (UID: "6d881eb0-5569-439f-859c-a86297985e58"). InnerVolumeSpecName "kube-api-access-slv9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:43:26.410413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.410380    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs" pod "6d881eb0-5569-439f-859c-a86297985e58" (UID: "6d881eb0-5569-439f-859c-a86297985e58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:43:26.473037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.473003    5534 generic.go:356] "Generic (PLEG): container finished" podID="6d881eb0-5569-439f-859c-a86297985e58" containerID="9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6" exitCode=0
Oct 30 17:43:26.473277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.473254    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq" event={"ID":"6d881eb0-5569-439f-859c-a86297985e58","Type":"ContainerDied","Data":"9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6"}
Oct 30 17:43:26.473359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.473320    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq" event={"ID":"6d881eb0-5569-439f-859c-a86297985e58","Type":"ContainerDied","Data":"bcec0a1441ff04f35c5951becd669d3dc5ce03437e5f1e283e6e7cd0f1d738fb"}
Oct 30 17:43:26.473435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.473356    5534 scope.go:122] "RemoveContainer" containerID="9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6"
Oct 30 17:43:26.473533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.473518    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"
Oct 30 17:43:26.485984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.485955    5534 scope.go:122] "RemoveContainer" containerID="9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6"
Oct 30 17:43:26.486412 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:43:26.486380    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6\": container with ID starting with 9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6 not found: ID does not exist" containerID="9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6"
Oct 30 17:43:26.486412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.486422    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6"} err="failed to get container status \"9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6\": rpc error: code = NotFound desc = could not find container \"9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6\": container with ID starting with 9b8332f42958c2dce17f462c1771928de3a42707105fe59d0cdb7683c25506e6 not found: ID does not exist"
Oct 30 17:43:26.488286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.488258    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d881eb0-5569-439f-859c-a86297985e58-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:26.488519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.488290    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:26.488519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.488329    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv9p\" (UniqueName: \"kubernetes.io/projected/6d881eb0-5569-439f-859c-a86297985e58-kube-api-access-slv9p\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:26.488519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.488342    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d881eb0-5569-439f-859c-a86297985e58-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:26.556699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.552611    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"]
Oct 30 17:43:26.556699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.556125    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:26.559682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.559622    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-802bda-867b6cd8f5-fq8pq"]
Oct 30 17:43:26.662254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.662219    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"]
Oct 30 17:43:26.666470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.666443    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-802bda-2vbxw"]
Oct 30 17:43:26.690835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:26.690733    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:27.211037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:27.211002    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:28.372397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:28.372366    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2de424-b8da-44c3-a954-5ca9e1545984" path="/var/lib/kubelet/pods/4b2de424-b8da-44c3-a954-5ca9e1545984/volumes"
Oct 30 17:43:28.372905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:28.372890    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d881eb0-5569-439f-859c-a86297985e58" path="/var/lib/kubelet/pods/6d881eb0-5569-439f-859c-a86297985e58/volumes"
Oct 30 17:43:28.862313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:28.862280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:30.735007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:30.734967    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:32.414056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:32.414026    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:43:32.490468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:32.490435    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:33.896534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:33.896478    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:34.071512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.071459    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:34.275894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.273084    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:34.481670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.480232    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:34.797497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.797455    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"]
Oct 30 17:43:34.798257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.798236    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6d881eb0-5569-439f-859c-a86297985e58" containerName="maas-api"
Oct 30 17:43:34.798330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.798268    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d881eb0-5569-439f-859c-a86297985e58" containerName="maas-api"
Oct 30 17:43:34.798435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.798426    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d881eb0-5569-439f-859c-a86297985e58" containerName="maas-api"
Oct 30 17:43:34.801966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.801950    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.826394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.826365    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"]
Oct 30 17:43:34.827167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.827139    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97w4\" (UniqueName: \"kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.827290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.827181    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.827355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.827341    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.892292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.892255    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:43:34.892476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.892302    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:43:34.929173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.929136    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.929581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.929319    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97w4\" (UniqueName: \"kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.929581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.929343    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.929986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.929968    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.931696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.931674    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:34.953736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:34.953705    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97w4\" (UniqueName: \"kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4\") pod \"maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:35.117043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:35.117009    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:35.439447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:35.439419    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"]
Oct 30 17:43:35.440003 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:35.439979    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7a50f1_96c7_4639_a707_56f5bfef08b3.slice/crio-735960d41c6f4eaae3f27bbded7786bdeb1794e330454c210c128f77d9a7dffc WatchSource:0}: Error finding container 735960d41c6f4eaae3f27bbded7786bdeb1794e330454c210c128f77d9a7dffc: Status 404 returned error can't find the container with id 735960d41c6f4eaae3f27bbded7786bdeb1794e330454c210c128f77d9a7dffc
Oct 30 17:43:35.574481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:35.574443    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74" event={"ID":"de7a50f1-96c7-4639-a707-56f5bfef08b3","Type":"ContainerStarted","Data":"735960d41c6f4eaae3f27bbded7786bdeb1794e330454c210c128f77d9a7dffc"}
Oct 30 17:43:36.470415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.470384    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:36.587222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.587183    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74" event={"ID":"de7a50f1-96c7-4639-a707-56f5bfef08b3","Type":"ContainerStarted","Data":"69cb7acd6f5ba73592196f67184a46ee5fd146d6e294c5873dae34005e21fd01"}
Oct 30 17:43:36.632904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.632854    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74" podStartSLOduration=2.632840622 podStartE2EDuration="2.632840622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:36.630936708 +0000 UTC m=+1822.392980523" watchObservedRunningTime="2026-10-30 17:43:36.632840622 +0000 UTC m=+1822.394884435"
Oct 30 17:43:36.672222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.672183    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"]
Oct 30 17:43:36.680329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.680302    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.696245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.696212    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-874b83-openshift-default-dockercfg-nmtqj\""
Oct 30 17:43:36.698593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.698565    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"]
Oct 30 17:43:36.757888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.757855    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759272    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdstz\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759389    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759447    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759508    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759535    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759582    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759611    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.759711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.759673    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861153    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861203    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdstz\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861271    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861311    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861371    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861396    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861493    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861522    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861559    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861803    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.861906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.861840    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.862097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.862061    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.862349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.862328    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.862906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.862882    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.864200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.864183    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.864371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.864353    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.903913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.903881    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:36.904221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:36.904198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdstz\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:37.068451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.068342    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:37.439617 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:37.439585    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76898535_a858_4ebb_8178_d0b97521ee76.slice/crio-8203016b86c3fdf8995385cef05db9053921105705c296dff4face0200152cdb WatchSource:0}: Error finding container 8203016b86c3fdf8995385cef05db9053921105705c296dff4face0200152cdb: Status 404 returned error can't find the container with id 8203016b86c3fdf8995385cef05db9053921105705c296dff4face0200152cdb
Oct 30 17:43:37.439820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.439800    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"]
Oct 30 17:43:37.443099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.443061    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:37.443214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.443151    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:37.443214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.443196    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:37.601575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.601519    5534 generic.go:356] "Generic (PLEG): container finished" podID="de7a50f1-96c7-4639-a707-56f5bfef08b3" containerID="69cb7acd6f5ba73592196f67184a46ee5fd146d6e294c5873dae34005e21fd01" exitCode=0
Oct 30 17:43:37.602138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.601772    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74" event={"ID":"de7a50f1-96c7-4639-a707-56f5bfef08b3","Type":"ContainerDied","Data":"69cb7acd6f5ba73592196f67184a46ee5fd146d6e294c5873dae34005e21fd01"}
Oct 30 17:43:37.603884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.603853    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" event={"ID":"76898535-a858-4ebb-8178-d0b97521ee76","Type":"ContainerStarted","Data":"76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76"}
Oct 30 17:43:37.604033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.603907    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" event={"ID":"76898535-a858-4ebb-8178-d0b97521ee76","Type":"ContainerStarted","Data":"8203016b86c3fdf8995385cef05db9053921105705c296dff4face0200152cdb"}
Oct 30 17:43:37.689456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.689330    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podStartSLOduration=1.689310455 podStartE2EDuration="1.689310455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:37.687893816 +0000 UTC m=+1823.449937629" watchObservedRunningTime="2026-10-30 17:43:37.689310455 +0000 UTC m=+1823.451354459"
Oct 30 17:43:37.709017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:37.708982    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:38.069212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.069123    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:38.070747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.070715    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:38.070877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.070777    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:38.173717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.173613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:38.604667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.603802    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"]
Oct 30 17:43:38.611674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.611584    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.640192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.640167    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ee9ff3-openshift-default-dockercfg-87c5x\""
Oct 30 17:43:38.648547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.648516    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"]
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705318    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705378    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qv2\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705458    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705589    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705635    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705720    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705744    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.705914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.706027    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.707448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.706169    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811033    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811553    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811618    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811662    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811840    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811928    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811969    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.812073    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qv2\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.816556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.812133    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.841323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.811473    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.843706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.842508    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:38.937713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:38.934037    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.045958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.011628    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qv2\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.092670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.092599    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.100665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.093373    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:39.104325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.103704    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:39.104325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.103757    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:39.104325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.104064    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.153691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.104796    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.214831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.195054    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.214831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.204438    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token\") pod \"e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") " pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.239881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.239849    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:39.244523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.244494    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:39.305681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.304880    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4\" (UniqueName: \"kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4\") pod \"de7a50f1-96c7-4639-a707-56f5bfef08b3\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") "
Oct 30 17:43:39.305681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.304978    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca\") pod \"de7a50f1-96c7-4639-a707-56f5bfef08b3\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") "
Oct 30 17:43:39.306598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.306568    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca\") pod \"de7a50f1-96c7-4639-a707-56f5bfef08b3\" (UID: \"de7a50f1-96c7-4639-a707-56f5bfef08b3\") "
Oct 30 17:43:39.308493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.308465    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca" pod "de7a50f1-96c7-4639-a707-56f5bfef08b3" (UID: "de7a50f1-96c7-4639-a707-56f5bfef08b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:43:39.310536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.310511    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4" pod "de7a50f1-96c7-4639-a707-56f5bfef08b3" (UID: "de7a50f1-96c7-4639-a707-56f5bfef08b3"). InnerVolumeSpecName "kube-api-access-z97w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:43:39.310689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.310662    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca" pod "de7a50f1-96c7-4639-a707-56f5bfef08b3" (UID: "de7a50f1-96c7-4639-a707-56f5bfef08b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:43:39.408795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.408758    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97w4\" (UniqueName: \"kubernetes.io/projected/de7a50f1-96c7-4639-a707-56f5bfef08b3-kube-api-access-z97w4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:39.408795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.408785    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:39.408795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.408795    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7a50f1-96c7-4639-a707-56f5bfef08b3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:39.644887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.644857    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"
Oct 30 17:43:39.644887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.644883    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74" event={"ID":"de7a50f1-96c7-4639-a707-56f5bfef08b3","Type":"ContainerDied","Data":"735960d41c6f4eaae3f27bbded7786bdeb1794e330454c210c128f77d9a7dffc"}
Oct 30 17:43:39.645418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.644927    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735960d41c6f4eaae3f27bbded7786bdeb1794e330454c210c128f77d9a7dffc"
Oct 30 17:43:39.694390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.694340    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:39.694615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.694596    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:39.694827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.694799    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:43:39.697868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:39.697722    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"]
Oct 30 17:43:40.086682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:40.085778    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:40.086682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:40.085842    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:40.677862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:40.677824    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" event={"ID":"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10","Type":"ContainerStarted","Data":"fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"}
Oct 30 17:43:40.678441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:40.678419    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" event={"ID":"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10","Type":"ContainerStarted","Data":"11fc9da4e7603be5bf1d526c810675b8fc78a76b0d1ff714af7f2e18a503ba2e"}
Oct 30 17:43:40.750949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:40.750899    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podStartSLOduration=2.750885785 podStartE2EDuration="2.750885785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:40.73543502 +0000 UTC m=+1826.497478832" watchObservedRunningTime="2026-10-30 17:43:40.750885785 +0000 UTC m=+1826.512929597"
Oct 30 17:43:40.996765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:40.996672    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:41.084668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.084200    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:41.084668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.084264    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:41.129518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.129481    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:41.245154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.245118    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:41.251368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.251296    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:41.272422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.271547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:41.697325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.697299    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:41.698105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:41.698077    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:43:42.036550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:42.036464    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"]
Oct 30 17:43:42.040080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:42.040053    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec015ffb-phnk8"]
Oct 30 17:43:42.074504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:42.074035    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:42.074504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:42.074099    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:42.398129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:42.398090    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fbb6cf-0168-4619-ac2f-c7335109e2fc" path="/var/lib/kubelet/pods/f5fbb6cf-0168-4619-ac2f-c7335109e2fc/volumes"
Oct 30 17:43:42.850883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:42.850848    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:43.071671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:43.070044    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:43.071671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:43.070108    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:43.308611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:43.308535    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:43.497957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:43.497923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:44.073181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:44.073143    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 17:43:44.073814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:44.073779    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 17:43:44.254459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:44.254422    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:44.500610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:44.500571    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:44.792716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:44.792618    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:45.075080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.074997    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:45.075549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.075413    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:45.076163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.076144    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:43:45.278069 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:43:45.278030    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:43:45.369380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.369347    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:45.611665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.609340    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:45.652727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.642754    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"]
Oct 30 17:43:45.652727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.644273    5534 cpu_manager.go:414] "containerMap: removing container" podUID="de7a50f1-96c7-4639-a707-56f5bfef08b3" containerName="revoke-keys"
Oct 30 17:43:45.652727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.644292    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7a50f1-96c7-4639-a707-56f5bfef08b3" containerName="revoke-keys"
Oct 30 17:43:45.652727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.644571    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7a50f1-96c7-4639-a707-56f5bfef08b3" containerName="revoke-keys"
Oct 30 17:43:45.667624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.667595    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:45.709786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.690188    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"]
Oct 30 17:43:45.711214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.710913    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972kc\" (UniqueName: \"kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc\") pod \"payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:45.711214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.711043    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:45.732665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.721183    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ee9ff3-dockercfg-2th2j\""
Oct 30 17:43:45.732665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.721857    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ee9ff3\""
Oct 30 17:43:45.812271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.812236    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972kc\" (UniqueName: \"kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc\") pod \"payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:45.812443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.812394    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:45.813226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.813204    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:45.838191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:45.838161    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972kc\" (UniqueName: \"kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc\") pod \"payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:46.072882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.072782    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:46.170751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.169596    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:46.410935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.400098    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"]
Oct 30 17:43:46.410935 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:46.406336    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea87737e_0ada_43ac_8950_bf844b867894.slice/crio-57a583e5683b0304208b7a95a56d0d18a4f9a09eaa2b62eafc58b1ec1c1e92c6 WatchSource:0}: Error finding container 57a583e5683b0304208b7a95a56d0d18a4f9a09eaa2b62eafc58b1ec1c1e92c6: Status 404 returned error can't find the container with id 57a583e5683b0304208b7a95a56d0d18a4f9a09eaa2b62eafc58b1ec1c1e92c6
Oct 30 17:43:46.639143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.639107    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"]
Oct 30 17:43:46.761308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.761200    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:46.799766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.799726    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg" event={"ID":"ea87737e-0ada-43ac-8950-bf844b867894","Type":"ContainerStarted","Data":"9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35"}
Oct 30 17:43:46.799937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.799788    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:46.799937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.799813    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg" event={"ID":"ea87737e-0ada-43ac-8950-bf844b867894","Type":"ContainerStarted","Data":"57a583e5683b0304208b7a95a56d0d18a4f9a09eaa2b62eafc58b1ec1c1e92c6"}
Oct 30 17:43:46.848485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.848409    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg" podStartSLOduration=1.84839114 podStartE2EDuration="1.84839114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:46.847810141 +0000 UTC m=+1832.609854005" watchObservedRunningTime="2026-10-30 17:43:46.84839114 +0000 UTC m=+1832.610434955"
Oct 30 17:43:46.940650 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:46.940619    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:48.052598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:48.052519    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"]
Oct 30 17:43:48.072434 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:48.072398    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd4e563-7qzgg"]
Oct 30 17:43:48.374247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:48.374213    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635e8b8c-0323-49ee-95c5-204b7baa0920" path="/var/lib/kubelet/pods/635e8b8c-0323-49ee-95c5-204b7baa0920/volumes"
Oct 30 17:43:48.496511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:48.496473    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:48.680731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:48.680404    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:50.303047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.303018    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:50.412944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.411473    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"]
Oct 30 17:43:50.419658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.419547    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:50.441498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.441464    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"]
Oct 30 17:43:50.452928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.452891    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24fb0d\""
Oct 30 17:43:50.484925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.484896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:50.526836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.526794    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbcc\" (UniqueName: \"kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc\") pod \"payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:50.527071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.527050    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:50.658673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.655347    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:50.658673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.655589    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbcc\" (UniqueName: \"kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc\") pod \"payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:50.732256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.725040    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:50.732256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.731734    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:50.803007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:50.802234    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbcc\" (UniqueName: \"kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc\") pod \"payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:51.035699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.035593    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:51.134667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.133878    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:51.377690 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:43:51.367217    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b8ffb3_e17f_434f_8a2d_c1caf4894554.slice/crio-40da5446e7f98b8ca50cd7b0e4e378c522d3d553dec96fe93d9c994f534c488f WatchSource:0}: Error finding container 40da5446e7f98b8ca50cd7b0e4e378c522d3d553dec96fe93d9c994f534c488f: Status 404 returned error can't find the container with id 40da5446e7f98b8ca50cd7b0e4e378c522d3d553dec96fe93d9c994f534c488f
Oct 30 17:43:51.377690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.370448    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"]
Oct 30 17:43:51.639589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.639497    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s" podUID="8a3a7310-8efd-426c-beed-da67d7bb5354" containerName="payload-pre-processing" containerID="cri-o://a2ff37051ec1aaa9722696b6901df0073bb988552ba98adb0c30a6fecac8f216" gracePeriod=25
Oct 30 17:43:51.902727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.902585    5534 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7310-8efd-426c-beed-da67d7bb5354" containerID="a2ff37051ec1aaa9722696b6901df0073bb988552ba98adb0c30a6fecac8f216" exitCode=0
Oct 30 17:43:51.902948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.902852    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s" event={"ID":"8a3a7310-8efd-426c-beed-da67d7bb5354","Type":"ContainerDied","Data":"a2ff37051ec1aaa9722696b6901df0073bb988552ba98adb0c30a6fecac8f216"}
Oct 30 17:43:51.917710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.906246    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" event={"ID":"64b8ffb3-e17f-434f-8a2d-c1caf4894554","Type":"ContainerStarted","Data":"d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53"}
Oct 30 17:43:51.917710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.906296    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" event={"ID":"64b8ffb3-e17f-434f-8a2d-c1caf4894554","Type":"ContainerStarted","Data":"40da5446e7f98b8ca50cd7b0e4e378c522d3d553dec96fe93d9c994f534c488f"}
Oct 30 17:43:51.917710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.906554    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:43:51.961978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:51.961951    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:43:52.004054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.003992    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" podStartSLOduration=2.003974756 podStartE2EDuration="2.003974756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:51.954519157 +0000 UTC m=+1837.716562970" watchObservedRunningTime="2026-10-30 17:43:52.003974756 +0000 UTC m=+1837.766018570"
Oct 30 17:43:52.097752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.097720    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj\" (UniqueName: \"kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj\") pod \"8a3a7310-8efd-426c-beed-da67d7bb5354\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") "
Oct 30 17:43:52.097984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.097841    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume\") pod \"8a3a7310-8efd-426c-beed-da67d7bb5354\" (UID: \"8a3a7310-8efd-426c-beed-da67d7bb5354\") "
Oct 30 17:43:52.098429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.098396    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume" pod "8a3a7310-8efd-426c-beed-da67d7bb5354" (UID: "8a3a7310-8efd-426c-beed-da67d7bb5354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:43:52.098926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.098900    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3a7310-8efd-426c-beed-da67d7bb5354-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:52.100888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.100853    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj" pod "8a3a7310-8efd-426c-beed-da67d7bb5354" (UID: "8a3a7310-8efd-426c-beed-da67d7bb5354"). InnerVolumeSpecName "kube-api-access-hvxhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:43:52.206773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.206582    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvxhj\" (UniqueName: \"kubernetes.io/projected/8a3a7310-8efd-426c-beed-da67d7bb5354-kube-api-access-hvxhj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:43:52.916859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.916832    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"
Oct 30 17:43:52.917328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.916868    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s" event={"ID":"8a3a7310-8efd-426c-beed-da67d7bb5354","Type":"ContainerDied","Data":"bd87f25f9da219410ed48ad33355293835de7bba51f03f0afd85af10cd5e6439"}
Oct 30 17:43:52.917328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.916935    5534 scope.go:122] "RemoveContainer" containerID="a2ff37051ec1aaa9722696b6901df0073bb988552ba98adb0c30a6fecac8f216"
Oct 30 17:43:52.970979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.970938    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"]
Oct 30 17:43:52.982907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:52.982880    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc4e96-78fc486b47-5zp9s"]
Oct 30 17:43:53.104764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:53.104730    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:54.374105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:54.374069    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3a7310-8efd-426c-beed-da67d7bb5354" path="/var/lib/kubelet/pods/8a3a7310-8efd-426c-beed-da67d7bb5354/volumes"
Oct 30 17:43:54.849937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:54.849905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:56.087444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:56.087413    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:56.499038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:56.499004    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:43:57.811600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:57.811570    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:43:58.457079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:43:58.457042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:00.309855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:00.309815    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:02.021011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:02.020977    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:02.145548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:02.145483    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:02.267513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:02.267480    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:02.920404 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:02.920378    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:44:04.097916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:04.097870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:05.894176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:05.894139    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:07.570316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:07.570195    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:09.386231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:09.386189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:11.108491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:11.108448    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:11.921447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:11.921411    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:12.078690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:12.072453    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:13.078086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:13.076182    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:15.030524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:15.030489    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:15.246708 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:44:15.246666    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:44:16.376486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:16.376455    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:16.504516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:16.504478    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"]
Oct 30 17:44:16.504743 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:16.504717    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9" podUID="d74d0783-6885-41b0-89d8-f5989412a3e3" containerName="maas-api" containerID="cri-o://bed0e4ae2a3231a2b551ce16df45e3907705ce1ee2a684f93c53d1d49a6092b1" gracePeriod=30
Oct 30 17:44:16.588782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:16.588706    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:17.062729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.062058    5534 scope.go:122] "RemoveContainer" containerID="e35908f523d73082728832c1055746ffaaff7e3752bec86835723d6425b2742e"
Oct 30 17:44:17.328255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.328156    5534 generic.go:356] "Generic (PLEG): container finished" podID="d74d0783-6885-41b0-89d8-f5989412a3e3" containerID="bed0e4ae2a3231a2b551ce16df45e3907705ce1ee2a684f93c53d1d49a6092b1" exitCode=0
Oct 30 17:44:17.328735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.328656    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9" event={"ID":"d74d0783-6885-41b0-89d8-f5989412a3e3","Type":"ContainerDied","Data":"bed0e4ae2a3231a2b551ce16df45e3907705ce1ee2a684f93c53d1d49a6092b1"}
Oct 30 17:44:17.335126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.335094    5534 scope.go:122] "RemoveContainer" containerID="58dcae5863c32c6d655526ddee9d301828508d237a27f5d31c88b11e4d0e0323"
Oct 30 17:44:17.360482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.360455    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:44:17.449467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.449431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:17.500970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.500934    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca\") pod \"d74d0783-6885-41b0-89d8-f5989412a3e3\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") "
Oct 30 17:44:17.501135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.501114    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl\" (UniqueName: \"kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl\") pod \"d74d0783-6885-41b0-89d8-f5989412a3e3\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") "
Oct 30 17:44:17.501249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.501234    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs\") pod \"d74d0783-6885-41b0-89d8-f5989412a3e3\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") "
Oct 30 17:44:17.501327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.501301    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca" pod "d74d0783-6885-41b0-89d8-f5989412a3e3" (UID: "d74d0783-6885-41b0-89d8-f5989412a3e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:44:17.501388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.501368    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\") pod \"d74d0783-6885-41b0-89d8-f5989412a3e3\" (UID: \"d74d0783-6885-41b0-89d8-f5989412a3e3\") "
Oct 30 17:44:17.502239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.502213    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d0783-6885-41b0-89d8-f5989412a3e3-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:17.503718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.503695    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs" pod "d74d0783-6885-41b0-89d8-f5989412a3e3" (UID: "d74d0783-6885-41b0-89d8-f5989412a3e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:44:17.503870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.503853    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls" pod "d74d0783-6885-41b0-89d8-f5989412a3e3" (UID: "d74d0783-6885-41b0-89d8-f5989412a3e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:44:17.505581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.505553    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl" pod "d74d0783-6885-41b0-89d8-f5989412a3e3" (UID: "d74d0783-6885-41b0-89d8-f5989412a3e3"). InnerVolumeSpecName "kube-api-access-4w5kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:17.562674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.554224    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"]
Oct 30 17:44:17.573876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.573843    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466900-pgpjm"]
Oct 30 17:44:17.604435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.604402    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5kl\" (UniqueName: \"kubernetes.io/projected/d74d0783-6885-41b0-89d8-f5989412a3e3-kube-api-access-4w5kl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:17.604435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.604435    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:17.604839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.604452    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d0783-6885-41b0-89d8-f5989412a3e3-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:17.618862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.618830    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:17.737072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:17.737041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:18.339548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:18.339522    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"
Oct 30 17:44:18.339780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:18.339562    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9" event={"ID":"d74d0783-6885-41b0-89d8-f5989412a3e3","Type":"ContainerDied","Data":"61ec0c10ef9bc0ff0d826ce31546a842dd9675ae974b426a1ad3c19bdf04c3d4"}
Oct 30 17:44:18.339780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:18.339625    5534 scope.go:122] "RemoveContainer" containerID="bed0e4ae2a3231a2b551ce16df45e3907705ce1ee2a684f93c53d1d49a6092b1"
Oct 30 17:44:18.372815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:18.372783    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309fff25-a695-4968-8987-bd2f8166d0a1" path="/var/lib/kubelet/pods/309fff25-a695-4968-8987-bd2f8166d0a1/volumes"
Oct 30 17:44:18.393404 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:18.393376    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"]
Oct 30 17:44:18.406170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:18.406146    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466900-c99899c59-swpc9"]
Oct 30 17:44:19.515844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:19.515795    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:20.375792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:20.375753    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74d0783-6885-41b0-89d8-f5989412a3e3" path="/var/lib/kubelet/pods/d74d0783-6885-41b0-89d8-f5989412a3e3/volumes"
Oct 30 17:44:21.411333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:21.411299    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:22.380597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:22.380562    5534 generic.go:356] "Generic (PLEG): container finished" podID="74ccb873-6839-424e-b006-d07d6bc224a9" containerID="1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f" exitCode=0
Oct 30 17:44:22.380841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:22.380634    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" event={"ID":"74ccb873-6839-424e-b006-d07d6bc224a9","Type":"ContainerDied","Data":"1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"}
Oct 30 17:44:22.381102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:22.381084    5534 scope.go:122] "RemoveContainer" containerID="1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"
Oct 30 17:44:22.382866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:22.382840    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:44:22.382959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:22.382905    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:44:22.382959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:22.382936    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:44:23.112148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:23.112116    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:23.391780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:23.391632    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" event={"ID":"74ccb873-6839-424e-b006-d07d6bc224a9","Type":"ContainerStarted","Data":"f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d"}
Oct 30 17:44:24.028636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:24.028605    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:44:24.029739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:24.029712    5534 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 17:44:24.029874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:24.029768    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 17:44:24.839830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:24.823560    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:25.029681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.029621    5534 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 17:44:25.030020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.029986    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 17:44:25.546497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.546458    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:25.694192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.694162    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"]
Oct 30 17:44:25.694671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.694619    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" containerID="cri-o://8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d" gracePeriod=30
Oct 30 17:44:25.728629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.728541    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:25.732303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.732264    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"]
Oct 30 17:44:25.750981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.750941    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49e155-hdpc5"]
Oct 30 17:44:25.945445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:25.945406    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:26.022887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:26.022848    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"]
Oct 30 17:44:26.023383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:26.023347    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy" containerID="cri-o://2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d" gracePeriod=30
Oct 30 17:44:26.029094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:26.029065    5534 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 17:44:26.029301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:26.029274    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 17:44:26.204971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:26.204856    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:26.393119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:26.393083    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61184f05-aa07-444a-bc0e-fd6a0f278fdd" path="/var/lib/kubelet/pods/61184f05-aa07-444a-bc0e-fd6a0f278fdd/volumes"
Oct 30 17:44:27.007867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:27.007834    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:27.029014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:27.028985    5534 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 17:44:27.029148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:27.029031    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 17:44:27.239082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:27.239045    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:27.880670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:27.867138    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:44:28.028409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:28.028376    5534 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 17:44:28.028967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:28.028437    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 17:44:28.089693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:28.089443    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:28.328252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:28.328146    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:28.515706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:28.515670    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:29.155412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:29.155376    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:44:29.157407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:29.157383    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:44:30.314593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:30.314562    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:30.536895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:30.536856    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:44:30.537078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:30.536939    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:44:30.629592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:30.629537    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:31.364442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.364410    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:44:31.470096 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.470061    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:44:31.478557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478506    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478579    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478606    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478706    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478733    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478776    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478840    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478883    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.478986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.478904    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket\") pod \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\" (UID: \"1c62976b-8f4a-4b0d-929c-7dcaf375911c\") "
Oct 30 17:44:31.479584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.479557    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.481230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.480813    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.481230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.480867    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.481230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.480899    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.481907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.481885    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:44:31.484496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.484462    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "kube-api-access-t7qjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:31.484929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.484904    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.486050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.486017    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:31.488127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.488102    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo" pod "1c62976b-8f4a-4b0d-929c-7dcaf375911c" (UID: "1c62976b-8f4a-4b0d-929c-7dcaf375911c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:44:31.539078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.539041    5534 generic.go:356] "Generic (PLEG): container finished" podID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerID="8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d" exitCode=0
Oct 30 17:44:31.539271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.539092    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"
Oct 30 17:44:31.539271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.539127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" event={"ID":"3c435b47-b991-46a7-a936-97c7b25fd6ba","Type":"ContainerDied","Data":"8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d"}
Oct 30 17:44:31.539271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.539184    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c" event={"ID":"3c435b47-b991-46a7-a936-97c7b25fd6ba","Type":"ContainerDied","Data":"8de45dc2be3af66328c30ab7e2c872b922ded93aae9b3966eae28b387d22e4a4"}
Oct 30 17:44:31.539271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.539216    5534 scope.go:122] "RemoveContainer" containerID="8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d"
Oct 30 17:44:31.540857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.540836    5534 generic.go:356] "Generic (PLEG): container finished" podID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerID="2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d" exitCode=0
Oct 30 17:44:31.540955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.540926    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"
Oct 30 17:44:31.541021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.540998    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" event={"ID":"1c62976b-8f4a-4b0d-929c-7dcaf375911c","Type":"ContainerDied","Data":"2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d"}
Oct 30 17:44:31.541068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.541054    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm" event={"ID":"1c62976b-8f4a-4b0d-929c-7dcaf375911c","Type":"ContainerDied","Data":"c1b494d83080383d259bca0731b0915f18c95b268d54cce27eaf4a0c1aea8c4f"}
Oct 30 17:44:31.550460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.550440    5534 scope.go:122] "RemoveContainer" containerID="8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d"
Oct 30 17:44:31.550778 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:44:31.550761    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d\": container with ID starting with 8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d not found: ID does not exist" containerID="8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d"
Oct 30 17:44:31.550843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.550784    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d"} err="failed to get container status \"8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d\": rpc error: code = NotFound desc = could not find container \"8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d\": container with ID starting with 8a335df988cbfe3ba5c9713a87274393540fae7fbe283bf6564a349a18cbc50d not found: ID does not exist"
Oct 30 17:44:31.550843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.550802    5534 scope.go:122] "RemoveContainer" containerID="2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d"
Oct 30 17:44:31.566247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.565911    5534 scope.go:122] "RemoveContainer" containerID="2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d"
Oct 30 17:44:31.566576 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:44:31.566490    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d\": container with ID starting with 2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d not found: ID does not exist" containerID="2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d"
Oct 30 17:44:31.566576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.566528    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d"} err="failed to get container status \"2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d\": rpc error: code = NotFound desc = could not find container \"2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d\": container with ID starting with 2755affe4fce93136adcc3b9a2abab68870cd9c5f0bb9b14cca86109fcb84f8d not found: ID does not exist"
Oct 30 17:44:31.580618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580543    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.580789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580667    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.580789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580751    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.580919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580847    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.580919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580890    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.581088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580943    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.581088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.580984    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.581215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.581194    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.581402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.581364    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.581716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.581635    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.581796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.581748    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.581957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.581828    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token\") pod \"3c435b47-b991-46a7-a936-97c7b25fd6ba\" (UID: \"3c435b47-b991-46a7-a936-97c7b25fd6ba\") "
Oct 30 17:44:31.581957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.581934    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.582224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.582204    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.582588    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.582614    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.582628    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583081    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583108    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583133    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583147    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qjj\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-kube-api-access-t7qjj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583162    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583178    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583194    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583206    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583220    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c435b47-b991-46a7-a936-97c7b25fd6ba-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583234    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c62976b-8f4a-4b0d-929c-7dcaf375911c-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.583700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.583247    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c62976b-8f4a-4b0d-929c-7dcaf375911c-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.584417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.584235    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "kube-api-access-7mm5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:31.586496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.586469    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:31.592030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.592000    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:44:31.594458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.594430    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy" pod "3c435b47-b991-46a7-a936-97c7b25fd6ba" (UID: "3c435b47-b991-46a7-a936-97c7b25fd6ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:31.659567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.652060    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"]
Oct 30 17:44:31.679720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.679684    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-wgpcm"]
Oct 30 17:44:31.685773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.685741    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.685773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.685777    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.685965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.685793    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.685965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.685808    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mm5s\" (UniqueName: \"kubernetes.io/projected/3c435b47-b991-46a7-a936-97c7b25fd6ba-kube-api-access-7mm5s\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:31.893170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.893132    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"]
Oct 30 17:44:31.899184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:31.899155    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-bw94c"]
Oct 30 17:44:32.384167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:32.384128    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" path="/var/lib/kubelet/pods/1c62976b-8f4a-4b0d-929c-7dcaf375911c/volumes"
Oct 30 17:44:32.385676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:32.385630    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" path="/var/lib/kubelet/pods/3c435b47-b991-46a7-a936-97c7b25fd6ba/volumes"
Oct 30 17:44:32.448484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:32.448450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:32.620975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:32.620892    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:33.738599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:33.738567    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:34.056730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:34.056205    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:34.179167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:34.179131    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:34.483369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:34.483332    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:34.993277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:34.993244    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:35.360953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:35.360923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:35.968737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:35.964240    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:36.185425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:36.185387    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:36.311251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:36.311128    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:36.434576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:36.434537    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:37.256527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:37.256494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:38.366491 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:38.366456    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:39.271791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:39.271747    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:39.914286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:39.914232    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:40.114873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:40.114840    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:41.614209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.614182    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:41.746770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.746713    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:41.836040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.836008    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"]
Oct 30 17:44:41.838611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.838584    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d74d0783-6885-41b0-89d8-f5989412a3e3" containerName="maas-api"
Oct 30 17:44:41.838826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.838810    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74d0783-6885-41b0-89d8-f5989412a3e3" containerName="maas-api"
Oct 30 17:44:41.838937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.838926    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8a3a7310-8efd-426c-beed-da67d7bb5354" containerName="payload-pre-processing"
Oct 30 17:44:41.839014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.839003    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3a7310-8efd-426c-beed-da67d7bb5354" containerName="payload-pre-processing"
Oct 30 17:44:41.839099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.839090    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy"
Oct 30 17:44:41.839167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.839157    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy"
Oct 30 17:44:41.839367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.839355    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy"
Oct 30 17:44:41.839447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.839437    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy"
Oct 30 17:44:41.839885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.839867    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74d0783-6885-41b0-89d8-f5989412a3e3" containerName="maas-api"
Oct 30 17:44:41.840035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.840022    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3a7310-8efd-426c-beed-da67d7bb5354" containerName="payload-pre-processing"
Oct 30 17:44:41.840180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.840167    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c62976b-8f4a-4b0d-929c-7dcaf375911c" containerName="istio-proxy"
Oct 30 17:44:41.840300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.840289    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c435b47-b991-46a7-a936-97c7b25fd6ba" containerName="istio-proxy"
Oct 30 17:44:41.851526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.851500    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:41.865263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.865194    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"]
Oct 30 17:44:41.870874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.870843    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-488c67\""
Oct 30 17:44:41.903824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.903776    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:41.904320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.904301    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"]
Oct 30 17:44:41.910178 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.910155    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:41.917977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.917947    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgw9q\" (UniqueName: \"kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q\") pod \"payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:41.918125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.918063    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:41.928215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.928192    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-488c67-dockercfg-w2nds\""
Oct 30 17:44:41.933522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:41.933495    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"]
Oct 30 17:44:42.034829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.034785    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:42.035039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.034939    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brgm\" (UniqueName: \"kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm\") pod \"payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.035175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.035156    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.035256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.035194    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgw9q\" (UniqueName: \"kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q\") pod \"payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:42.036478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.036444    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:42.172872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.140764    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brgm\" (UniqueName: \"kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm\") pod \"payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.174791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.174758    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.175725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.175698    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.187465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.184389    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgw9q\" (UniqueName: \"kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q\") pod \"payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:42.187465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.184785    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:42.197530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.196562    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brgm\" (UniqueName: \"kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm\") pod \"payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.225896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.225856    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.551737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.551700    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"]
Oct 30 17:44:42.552108 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:44:42.552086    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25dd3c0c_58f4_4e48_bffa_96a9a14c5157.slice/crio-1894ad5eb03d69de9e673895de13e391f6030c5353a8612a5d1c64e161d02dbb WatchSource:0}: Error finding container 1894ad5eb03d69de9e673895de13e391f6030c5353a8612a5d1c64e161d02dbb: Status 404 returned error can't find the container with id 1894ad5eb03d69de9e673895de13e391f6030c5353a8612a5d1c64e161d02dbb
Oct 30 17:44:42.578824 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:44:42.578786    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26c387f_250d_4e59_8106_b25975fc9f3f.slice/crio-49c694bc87618dc700542e846f4f0da1d5ea266db5734442dce645d1c950d225 WatchSource:0}: Error finding container 49c694bc87618dc700542e846f4f0da1d5ea266db5734442dce645d1c950d225: Status 404 returned error can't find the container with id 49c694bc87618dc700542e846f4f0da1d5ea266db5734442dce645d1c950d225
Oct 30 17:44:42.584057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.584028    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"]
Oct 30 17:44:42.718841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.718805    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" event={"ID":"25dd3c0c-58f4-4e48-bffa-96a9a14c5157","Type":"ContainerStarted","Data":"f6f19060f2a0dcb04ff5c0c7ba1bc11e8c634b57f4db3c0a18413926a954c922"}
Oct 30 17:44:42.719301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.718873    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:42.719301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.718897    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" event={"ID":"25dd3c0c-58f4-4e48-bffa-96a9a14c5157","Type":"ContainerStarted","Data":"1894ad5eb03d69de9e673895de13e391f6030c5353a8612a5d1c64e161d02dbb"}
Oct 30 17:44:42.720515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.720490    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" event={"ID":"d26c387f-250d-4e59-8106-b25975fc9f3f","Type":"ContainerStarted","Data":"356efe66a8b366c2d1ca3ca9ad1bb59ea1aa71b917f43e436809305d61e6a333"}
Oct 30 17:44:42.720619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.720530    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" event={"ID":"d26c387f-250d-4e59-8106-b25975fc9f3f","Type":"ContainerStarted","Data":"49c694bc87618dc700542e846f4f0da1d5ea266db5734442dce645d1c950d225"}
Oct 30 17:44:42.720619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.720552    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:42.767760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.767697    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" podStartSLOduration=1.767678283 podStartE2EDuration="1.767678283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:42.762344875 +0000 UTC m=+1888.524388691" watchObservedRunningTime="2026-10-30 17:44:42.767678283 +0000 UTC m=+1888.529722101"
Oct 30 17:44:42.807599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:42.807497    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" podStartSLOduration=1.807481864 podStartE2EDuration="1.807481864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:42.80419407 +0000 UTC m=+1888.566237885" watchObservedRunningTime="2026-10-30 17:44:42.807481864 +0000 UTC m=+1888.569525676"
Oct 30 17:44:43.499868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:43.499828    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:43.681926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:43.681888    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:45.255887 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:44:45.255844    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:44:45.631336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:45.631304    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:46.642112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:46.642074    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"]
Oct 30 17:44:46.642666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:46.642327    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy" containerID="cri-o://2ca5132d0c6db7796a12340bb5975ad003e7ef3b570d0a55bd4bf2261fb2ce41" gracePeriod=30
Oct 30 17:44:46.685634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:46.685607    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:47.218587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.218543    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"]
Oct 30 17:44:47.223533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.223510    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.246776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.246746    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:44:47.246921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.246872    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:44:47.265441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.265410    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"]
Oct 30 17:44:47.363201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.363151    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.363410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.363209    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gz55\" (UniqueName: \"kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.363410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.363360    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.465790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.465753    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.466016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.465822    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gz55\" (UniqueName: \"kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.466016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.465986    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.466935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.466912    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.469012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.468951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.490198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.490170    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gz55\" (UniqueName: \"kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55\") pod \"maas-api-revoke-keys-e2e-api-b-74762d-v2xdp\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.544667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.543893    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:47.579814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.579784    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:47.906219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:47.906181    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"]
Oct 30 17:44:48.827614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:48.827573    5534 generic.go:356] "Generic (PLEG): container finished" podID="453432ed-2dc8-461d-aca3-1a9c64ef8860" containerID="a707a57c47253203151a6e9bbb70ee62a87cb1dc4528dea6cb9643cdfdb54099" exitCode=0
Oct 30 17:44:48.827832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:48.827677    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp" event={"ID":"453432ed-2dc8-461d-aca3-1a9c64ef8860","Type":"ContainerDied","Data":"a707a57c47253203151a6e9bbb70ee62a87cb1dc4528dea6cb9643cdfdb54099"}
Oct 30 17:44:48.829411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:48.829384    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp" event={"ID":"453432ed-2dc8-461d-aca3-1a9c64ef8860","Type":"ContainerStarted","Data":"447961f01d655e844f4c42c3560179d677908b61ff77cc77f42c57efac778bd9"}
Oct 30 17:44:49.314671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:49.314615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:50.012157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.012127    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:50.115717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.115684    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca\") pod \"453432ed-2dc8-461d-aca3-1a9c64ef8860\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") "
Oct 30 17:44:50.115941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.115763    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55\" (UniqueName: \"kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55\") pod \"453432ed-2dc8-461d-aca3-1a9c64ef8860\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") "
Oct 30 17:44:50.115941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.115828    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca\") pod \"453432ed-2dc8-461d-aca3-1a9c64ef8860\" (UID: \"453432ed-2dc8-461d-aca3-1a9c64ef8860\") "
Oct 30 17:44:50.116172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.116144    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca" pod "453432ed-2dc8-461d-aca3-1a9c64ef8860" (UID: "453432ed-2dc8-461d-aca3-1a9c64ef8860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:44:50.116613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.116574    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:50.118144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.118117    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca" pod "453432ed-2dc8-461d-aca3-1a9c64ef8860" (UID: "453432ed-2dc8-461d-aca3-1a9c64ef8860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:44:50.118330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.118308    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55" pod "453432ed-2dc8-461d-aca3-1a9c64ef8860" (UID: "453432ed-2dc8-461d-aca3-1a9c64ef8860"). InnerVolumeSpecName "kube-api-access-4gz55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:50.217948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.217912    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gz55\" (UniqueName: \"kubernetes.io/projected/453432ed-2dc8-461d-aca3-1a9c64ef8860-kube-api-access-4gz55\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:50.217948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.217944    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453432ed-2dc8-461d-aca3-1a9c64ef8860-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:50.492596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.492565    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:50.607973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.607939    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:50.719385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.719277    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:50.844385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.844354    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp" event={"ID":"453432ed-2dc8-461d-aca3-1a9c64ef8860","Type":"ContainerDied","Data":"447961f01d655e844f4c42c3560179d677908b61ff77cc77f42c57efac778bd9"}
Oct 30 17:44:50.844552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.844388    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"
Oct 30 17:44:50.844552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:50.844400    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="447961f01d655e844f4c42c3560179d677908b61ff77cc77f42c57efac778bd9"
Oct 30 17:44:51.629075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:51.629035    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:51.887720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:51.887501    5534 generic.go:356] "Generic (PLEG): container finished" podID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerID="2ca5132d0c6db7796a12340bb5975ad003e7ef3b570d0a55bd4bf2261fb2ce41" exitCode=0
Oct 30 17:44:51.887720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:51.887609    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" event={"ID":"fbc06e32-12ae-42a5-991a-2959bdd90cf3","Type":"ContainerDied","Data":"2ca5132d0c6db7796a12340bb5975ad003e7ef3b570d0a55bd4bf2261fb2ce41"}
Oct 30 17:44:51.909174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:51.891737    5534 scope.go:122] "RemoveContainer" containerID="0ce24f70b5c81d11e4c251526a8c98786c181189ddfa11036584bbb0d4601a86"
Oct 30 17:44:52.041576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.041552    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:44:52.151600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.151517    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.151600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.151566    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.151865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.151611    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.151865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.151771    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.151865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.151860    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.152057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152004    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.152057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152019    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:52.152175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152050    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:44:52.152175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152129    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.152175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152164    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.152336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152210    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy\") pod \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\" (UID: \"fbc06e32-12ae-42a5-991a-2959bdd90cf3\") "
Oct 30 17:44:52.152336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152279    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:52.152502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152472    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:52.152625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.152586    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:52.191615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.153140    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.191615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.153167    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.191615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.153181    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.191615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.153196    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.191615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.153209    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.193905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.192948    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:44:52.193905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.193721    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "kube-api-access-pnmch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:52.193905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.193841    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:44:52.194566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.194536    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token" pod "fbc06e32-12ae-42a5-991a-2959bdd90cf3" (UID: "fbc06e32-12ae-42a5-991a-2959bdd90cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:44:52.279668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.260974    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.279668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.261010    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnmch\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-kube-api-access-pnmch\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.279668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.261031    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.279668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.261045    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc06e32-12ae-42a5-991a-2959bdd90cf3-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:44:52.898338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.898299    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4" event={"ID":"fbc06e32-12ae-42a5-991a-2959bdd90cf3","Type":"ContainerDied","Data":"3baa1ec0e878d13188f5f25b27750b28aaf377f0ce38a208a63cca3d18180f40"}
Oct 30 17:44:52.898884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.898393    5534 scope.go:122] "RemoveContainer" containerID="2ca5132d0c6db7796a12340bb5975ad003e7ef3b570d0a55bd4bf2261fb2ce41"
Oct 30 17:44:52.898884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.898521    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"
Oct 30 17:44:52.992211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.992179    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"]
Oct 30 17:44:52.996364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:52.996334    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-466900-openshift-default-5489c88979-pfnx4"]
Oct 30 17:44:53.731501 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:53.731463    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:44:53.731680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:53.731537    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:44:53.771818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:53.769129    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:54.392478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:54.392378    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" path="/var/lib/kubelet/pods/fbc06e32-12ae-42a5-991a-2959bdd90cf3/volumes"
Oct 30 17:44:55.293111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:55.293082    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:55.634239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:55.634204    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:56.329867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:56.329826    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:57.565816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:57.565716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:57.684807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:57.684772    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"]
Oct 30 17:44:57.825604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:57.825516    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:57.948430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:57.948393    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:44:59.498436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:44:59.498403    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:00.151689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.150725    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"]
Oct 30 17:45:00.151954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.151931    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy"
Oct 30 17:45:00.152027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.151958    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy"
Oct 30 17:45:00.152027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.151988    5534 cpu_manager.go:414] "containerMap: removing container" podUID="453432ed-2dc8-461d-aca3-1a9c64ef8860" containerName="revoke-keys"
Oct 30 17:45:00.152027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.151998    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453432ed-2dc8-461d-aca3-1a9c64ef8860" containerName="revoke-keys"
Oct 30 17:45:00.152265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.152249    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy"
Oct 30 17:45:00.152337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.152273    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="453432ed-2dc8-461d-aca3-1a9c64ef8860" containerName="revoke-keys"
Oct 30 17:45:00.152337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.152296    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy"
Oct 30 17:45:00.192774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.189853    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"]
Oct 30 17:45:00.192774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.191035    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy"
Oct 30 17:45:00.192774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.191057    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc06e32-12ae-42a5-991a-2959bdd90cf3" containerName="istio-proxy"
Oct 30 17:45:00.193739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.193235    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.219832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.217160    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"]
Oct 30 17:45:00.223158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.223099    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.246823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.241350    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"]
Oct 30 17:45:00.246823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.243705    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.246823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.243796    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhwt\" (UniqueName: \"kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.246823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.243877    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.246823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.244000    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7j5\" (UniqueName: \"kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.265699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.252883    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.299772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.287950    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"]
Oct 30 17:45:00.303766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.303739    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.326467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.326437    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"]
Oct 30 17:45:00.335710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.327287    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.342323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.341039    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:45:00.345312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.345282    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfll\" (UniqueName: \"kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll\") pod \"maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.345540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.345523    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.346818    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347027    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhwt\" (UniqueName: \"kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347113    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347166    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5tv\" (UniqueName: \"kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347262    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7j5\" (UniqueName: \"kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347288    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347318    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347408    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.347536    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkvc\" (UniqueName: \"kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.352409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.348748    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.359935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.359909    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"]
Oct 30 17:45:00.360071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.359992    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.384398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.382128    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"]
Oct 30 17:45:00.387522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.384756    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.409723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.409613    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhwt\" (UniqueName: \"kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.417837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.417806    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7j5\" (UniqueName: \"kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.427047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.427012    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"]
Oct 30 17:45:00.427217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.427089    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"]
Oct 30 17:45:00.427217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.427174    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"]
Oct 30 17:45:00.427217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.427201    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"]
Oct 30 17:45:00.427575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.427227    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"]
Oct 30 17:45:00.448436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448406    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5tv\" (UniqueName: \"kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.448548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448455    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqmh\" (UniqueName: \"kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.448548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448504    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.448548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448521    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.448689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448552    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.448689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448613    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.448689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448654    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkvc\" (UniqueName: \"kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.448799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448699    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzh2\" (UniqueName: \"kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.448799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448729    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfll\" (UniqueName: \"kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll\") pod \"maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.448877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.448858    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.449495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.449474    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.449754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.449732    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.450071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.450045    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.455511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.455491    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"]
Oct 30 17:45:00.455652 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.455623    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.476452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.476422    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkvc\" (UniqueName: \"kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.476452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.476439    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfll\" (UniqueName: \"kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll\") pod \"maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.478013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.477993    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5tv\" (UniqueName: \"kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.483422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.483402    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"]
Oct 30 17:45:00.483511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.483445    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"]
Oct 30 17:45:00.483556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.483525    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.484237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.484216    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:00.488432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.488411    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"]
Oct 30 17:45:00.488549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.488495    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.493732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.493710    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"]
Oct 30 17:45:00.493859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.493754    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"]
Oct 30 17:45:00.493859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.493783    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"]
Oct 30 17:45:00.494083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.494060    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.498676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.498626    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"]
Oct 30 17:45:00.499213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.498696    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"]
Oct 30 17:45:00.499213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.498727    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"]
Oct 30 17:45:00.499213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.498751    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"]
Oct 30 17:45:00.499213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.498883    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.550143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.550107    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.550408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.550384    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.550479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.550443    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5ql\" (UniqueName: \"kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.550539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.550521    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.551401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551376    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.551541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551499    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndx7\" (UniqueName: \"kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7\") pod \"maas-api-key-cleanup-29889705-kk7st\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.551679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551553    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzh2\" (UniqueName: \"kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.551679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551609    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mld2r\" (UniqueName: \"kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.551807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551720    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.551807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551773    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.551941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551842    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65vw\" (UniqueName: \"kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.551941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.551894    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.552334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.552304    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889705-kk7st\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.552444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.552426    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqmh\" (UniqueName: \"kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.557397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.552984    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.557397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.552473    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jt2\" (UniqueName: \"kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2\") pod \"maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.596828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.576548    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:00.596828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.577290    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqmh\" (UniqueName: \"kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.596828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.588891    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzh2\" (UniqueName: \"kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.677370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.677333    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jt2\" (UniqueName: \"kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2\") pod \"maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.677519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.677443    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.677519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.677497    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.677657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.677543    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5ql\" (UniqueName: \"kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.678144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.678113    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndx7\" (UniqueName: \"kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7\") pod \"maas-api-key-cleanup-29889705-kk7st\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.678253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.678233    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mld2r\" (UniqueName: \"kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.678398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.678379    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.678485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.678468    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65vw\" (UniqueName: \"kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.678568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.678552    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.678731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.678694    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889705-kk7st\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.679807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.679415    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.679807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.679489    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.679807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.679700    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889705-kk7st\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.680354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.680325    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.680431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.680370    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.693241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.693021    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:00.716717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.709428    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jt2\" (UniqueName: \"kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2\") pod \"maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.716717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.715228    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5ql\" (UniqueName: \"kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.716717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.715744    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndx7\" (UniqueName: \"kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7\") pod \"maas-api-key-cleanup-29889705-kk7st\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.723492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.723295    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65vw\" (UniqueName: \"kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.723492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.723430    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mld2r\" (UniqueName: \"kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.734803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.734289    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:00.778952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.778711    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:00.790303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.790111    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:00.791464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.790599    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:00.825371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.824542    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"]
Oct 30 17:45:00.835308 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:00.835263    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f15f5e4_aa84_478e_9111_bff23ac5463a.slice/crio-5d36bc8084a69972ba73818d7614275d0725d676582eccb1ae21371687810e41 WatchSource:0}: Error finding container 5d36bc8084a69972ba73818d7614275d0725d676582eccb1ae21371687810e41: Status 404 returned error can't find the container with id 5d36bc8084a69972ba73818d7614275d0725d676582eccb1ae21371687810e41
Oct 30 17:45:00.852570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.851117    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:00.868247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.868057    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:00.883678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.880886    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:00.900442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.899919    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:00.941228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.941192    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:00.972414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:00.949277    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"]
Oct 30 17:45:01.120118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.108820    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" event={"ID":"6f15f5e4-aa84-478e-9111-bff23ac5463a","Type":"ContainerStarted","Data":"5d36bc8084a69972ba73818d7614275d0725d676582eccb1ae21371687810e41"}
Oct 30 17:45:01.120118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.110537    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerStarted","Data":"92d77b2d6354e5336284b0342b5cee65fa49287634002405f752a56dcbe54138"}
Oct 30 17:45:01.309632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.296983    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"]
Oct 30 17:45:01.310496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.310463    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"]
Oct 30 17:45:01.318447 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.317469    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6e8a5a_e240_4f63_8e30_f2d5b5295b8d.slice/crio-4caf2df334756fe6b2f0d1489b8a4924d9654e8d8b07a3521afaf94cc4ad2aa3 WatchSource:0}: Error finding container 4caf2df334756fe6b2f0d1489b8a4924d9654e8d8b07a3521afaf94cc4ad2aa3: Status 404 returned error can't find the container with id 4caf2df334756fe6b2f0d1489b8a4924d9654e8d8b07a3521afaf94cc4ad2aa3
Oct 30 17:45:01.399880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.399673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:01.430142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.430111    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"]
Oct 30 17:45:01.434009 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.433968    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3b6fe8_94ca_444d_90ef_78aa4bbccf58.slice/crio-8c18eddfb319285ee3e5f3c47b7a2080075d3dc412f815874dd0efe16b920dd2 WatchSource:0}: Error finding container 8c18eddfb319285ee3e5f3c47b7a2080075d3dc412f815874dd0efe16b920dd2: Status 404 returned error can't find the container with id 8c18eddfb319285ee3e5f3c47b7a2080075d3dc412f815874dd0efe16b920dd2
Oct 30 17:45:01.497780 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.497736    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a5d82b_7418_42a6_8265_1a889f3ebecf.slice/crio-5a94fb39b01d7500ce141b8fddeeb925fd7f1ecfee0a9861a9630ee4f75d1781 WatchSource:0}: Error finding container 5a94fb39b01d7500ce141b8fddeeb925fd7f1ecfee0a9861a9630ee4f75d1781: Status 404 returned error can't find the container with id 5a94fb39b01d7500ce141b8fddeeb925fd7f1ecfee0a9861a9630ee4f75d1781
Oct 30 17:45:01.498687 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.498600    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85827e4b_8f28_4b11_a1bb_b8b78dd94d43.slice/crio-22f3974b33015609add8060cf1c576efce1110ac43c4409256044359c2f208be WatchSource:0}: Error finding container 22f3974b33015609add8060cf1c576efce1110ac43c4409256044359c2f208be: Status 404 returned error can't find the container with id 22f3974b33015609add8060cf1c576efce1110ac43c4409256044359c2f208be
Oct 30 17:45:01.510202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.506258    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"]
Oct 30 17:45:01.510202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.509816    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"]
Oct 30 17:45:01.604276 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.604230    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e9bdd1_ed4d_493a_b112_e24721540858.slice/crio-bbfa22129e310c07e7e03b19c8ef147f053fe08d6ec0cc7cf896f43542a38712 WatchSource:0}: Error finding container bbfa22129e310c07e7e03b19c8ef147f053fe08d6ec0cc7cf896f43542a38712: Status 404 returned error can't find the container with id bbfa22129e310c07e7e03b19c8ef147f053fe08d6ec0cc7cf896f43542a38712
Oct 30 17:45:01.605047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.605021    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"]
Oct 30 17:45:01.654778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.654747    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"]
Oct 30 17:45:01.656993 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.656965    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d7ba6a_9ead_4270_a755_2ccbec88707e.slice/crio-0e128c969b38bf562ba1dd5125f20ac61f2962c36e125e314eb2373c2740c012 WatchSource:0}: Error finding container 0e128c969b38bf562ba1dd5125f20ac61f2962c36e125e314eb2373c2740c012: Status 404 returned error can't find the container with id 0e128c969b38bf562ba1dd5125f20ac61f2962c36e125e314eb2373c2740c012
Oct 30 17:45:01.701319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.701281    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"]
Oct 30 17:45:01.715631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.715597    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"]
Oct 30 17:45:01.716315 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.716289    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac47d37_9668_42b9_a8c7_e24eb5d1afa9.slice/crio-ab2e9dea85a14258fcc8362fa1498a5acd728b68981bd657dda3b80281a3ef70 WatchSource:0}: Error finding container ab2e9dea85a14258fcc8362fa1498a5acd728b68981bd657dda3b80281a3ef70: Status 404 returned error can't find the container with id ab2e9dea85a14258fcc8362fa1498a5acd728b68981bd657dda3b80281a3ef70
Oct 30 17:45:01.728547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:01.728519    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"]
Oct 30 17:45:01.732462 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:01.732397    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1f04ed_4c86_4c75_bdc2_20aa12564d40.slice/crio-8e7be96064dbfbaa8c8c5fe185d4a6c1a5bda58cbed38d8c08b77f252fa0cc8b WatchSource:0}: Error finding container 8e7be96064dbfbaa8c8c5fe185d4a6c1a5bda58cbed38d8c08b77f252fa0cc8b: Status 404 returned error can't find the container with id 8e7be96064dbfbaa8c8c5fe185d4a6c1a5bda58cbed38d8c08b77f252fa0cc8b
Oct 30 17:45:02.011146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.010778    5534 patch_prober.go:29] interesting pod/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": read tcp 10.133.0.2:36320->10.133.0.187:15021: read: connection reset by peer" start-of-body=
Oct 30 17:45:02.011146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.010835    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": read tcp 10.133.0.2:36320->10.133.0.187:15021: read: connection reset by peer"
Oct 30 17:45:02.122417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.122378    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" event={"ID":"6f15f5e4-aa84-478e-9111-bff23ac5463a","Type":"ContainerStarted","Data":"fbadda332c790a221de7264292553be40b88cbcd4ccca714ef69d7b0b2144f3e"}
Oct 30 17:45:02.124218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.124188    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerStarted","Data":"42368e17030ac26d1d0e9f445c2fa53229ec662447f43c49fffdf32fb1cc8e9a"}
Oct 30 17:45:02.124359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.124241    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerStarted","Data":"4caf2df334756fe6b2f0d1489b8a4924d9654e8d8b07a3521afaf94cc4ad2aa3"}
Oct 30 17:45:02.125618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.125594    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerStarted","Data":"8e7be96064dbfbaa8c8c5fe185d4a6c1a5bda58cbed38d8c08b77f252fa0cc8b"}
Oct 30 17:45:02.127021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.126993    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerStarted","Data":"ab2e9dea85a14258fcc8362fa1498a5acd728b68981bd657dda3b80281a3ef70"}
Oct 30 17:45:02.128546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.128522    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerStarted","Data":"dca2f091e01197ab80b7082985698ef8159248bd1853ce147690bc4c6ab5c44e"}
Oct 30 17:45:02.129822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.129800    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" event={"ID":"f2a63f21-1afe-421e-9b25-91c082d51442","Type":"ContainerStarted","Data":"17af438a12b1184aafebc5835c6ae1ecf172ffb43c7cd0f308018755eef18097"}
Oct 30 17:45:02.131116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.131093    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerStarted","Data":"0e128c969b38bf562ba1dd5125f20ac61f2962c36e125e314eb2373c2740c012"}
Oct 30 17:45:02.132436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.132406    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerStarted","Data":"8c18eddfb319285ee3e5f3c47b7a2080075d3dc412f815874dd0efe16b920dd2"}
Oct 30 17:45:02.134087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.134065    5534 generic.go:356] "Generic (PLEG): container finished" podID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerID="f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a" exitCode=0
Oct 30 17:45:02.134185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.134137    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" event={"ID":"105ab0a0-4585-4c94-8eb6-c789cd5b00a1","Type":"ContainerDied","Data":"f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"}
Oct 30 17:45:02.134538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.134523    5534 scope.go:122] "RemoveContainer" containerID="f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"
Oct 30 17:45:02.135817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.135792    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerStarted","Data":"6c0cdffda10ddf3ec14375d42136385142ac9aed1d67286992ba6abaea7585f9"}
Oct 30 17:45:02.135921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.135837    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerStarted","Data":"ee6b273a27d59c57cadba26839e99c5293d743b312ae1dc031bedcc184274dda"}
Oct 30 17:45:02.136529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.136502    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:02.136682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.136588    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:02.136682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.136659    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:02.137471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.137451    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerStarted","Data":"5a94fb39b01d7500ce141b8fddeeb925fd7f1ecfee0a9861a9630ee4f75d1781"}
Oct 30 17:45:02.138981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.138959    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" event={"ID":"85827e4b-8f28-4b11-a1bb-b8b78dd94d43","Type":"ContainerStarted","Data":"22f3974b33015609add8060cf1c576efce1110ac43c4409256044359c2f208be"}
Oct 30 17:45:02.140396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.140350    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerStarted","Data":"bbfa22129e310c07e7e03b19c8ef147f053fe08d6ec0cc7cf896f43542a38712"}
Oct 30 17:45:02.163674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.163575    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" podStartSLOduration=2.163556964 podStartE2EDuration="2.163556964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:02.161238801 +0000 UTC m=+1907.923282618" watchObservedRunningTime="2026-10-30 17:45:02.163556964 +0000 UTC m=+1907.925600778"
Oct 30 17:45:02.311068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.311019    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" podStartSLOduration=2.311005484 podStartE2EDuration="2.311005484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:02.308193391 +0000 UTC m=+1908.070237205" watchObservedRunningTime="2026-10-30 17:45:02.311005484 +0000 UTC m=+1908.073049297"
Oct 30 17:45:02.318999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.312299    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" podStartSLOduration=2.31228415 podStartE2EDuration="2.31228415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:02.260010511 +0000 UTC m=+1908.022054351" watchObservedRunningTime="2026-10-30 17:45:02.31228415 +0000 UTC m=+1908.074327964"
Oct 30 17:45:02.352057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.352010    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" podStartSLOduration=2.351995417 podStartE2EDuration="2.351995417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:02.350216303 +0000 UTC m=+1908.112260116" watchObservedRunningTime="2026-10-30 17:45:02.351995417 +0000 UTC m=+1908.114039230"
Oct 30 17:45:02.685257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:02.685224    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4" podUID="78ba9c34-8c81-4b68-90dc-baa751b5cb93" containerName="payload-processing" containerID="cri-o://5d732808a209c23ace4b4acfc62d13ab94ca79b5cfa8be3ae7dca9a81f538677" gracePeriod=25
Oct 30 17:45:03.150156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.150119    5534 generic.go:356] "Generic (PLEG): container finished" podID="95e9bdd1-ed4d-493a-b112-e24721540858" containerID="9cf2377fab03bd8752a9e0f828c62b0f1652d6b74fc8f631c6d5b3fe32f449a1" exitCode=60
Oct 30 17:45:03.150300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.150198    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerDied","Data":"9cf2377fab03bd8752a9e0f828c62b0f1652d6b74fc8f631c6d5b3fe32f449a1"}
Oct 30 17:45:03.150538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.150516    5534 scope.go:122] "RemoveContainer" containerID="9cf2377fab03bd8752a9e0f828c62b0f1652d6b74fc8f631c6d5b3fe32f449a1"
Oct 30 17:45:03.152015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.151993    5534 generic.go:356] "Generic (PLEG): container finished" podID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerID="fbadda332c790a221de7264292553be40b88cbcd4ccca714ef69d7b0b2144f3e" exitCode=60
Oct 30 17:45:03.152131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.152072    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" event={"ID":"6f15f5e4-aa84-478e-9111-bff23ac5463a","Type":"ContainerDied","Data":"fbadda332c790a221de7264292553be40b88cbcd4ccca714ef69d7b0b2144f3e"}
Oct 30 17:45:03.152278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.152258    5534 scope.go:122] "RemoveContainer" containerID="fbadda332c790a221de7264292553be40b88cbcd4ccca714ef69d7b0b2144f3e"
Oct 30 17:45:03.192689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.191692    5534 generic.go:356] "Generic (PLEG): container finished" podID="78ba9c34-8c81-4b68-90dc-baa751b5cb93" containerID="5d732808a209c23ace4b4acfc62d13ab94ca79b5cfa8be3ae7dca9a81f538677" exitCode=0
Oct 30 17:45:03.192689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.191780    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4" event={"ID":"78ba9c34-8c81-4b68-90dc-baa751b5cb93","Type":"ContainerDied","Data":"5d732808a209c23ace4b4acfc62d13ab94ca79b5cfa8be3ae7dca9a81f538677"}
Oct 30 17:45:03.192689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.191829    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4" event={"ID":"78ba9c34-8c81-4b68-90dc-baa751b5cb93","Type":"ContainerDied","Data":"738af6d52cc88e04ad9ce02e51d10071e70b2900eb01b510c24fe99151778034"}
Oct 30 17:45:03.192689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.191853    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738af6d52cc88e04ad9ce02e51d10071e70b2900eb01b510c24fe99151778034"
Oct 30 17:45:03.197967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.196162    5534 generic.go:356] "Generic (PLEG): container finished" podID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerID="42368e17030ac26d1d0e9f445c2fa53229ec662447f43c49fffdf32fb1cc8e9a" exitCode=60
Oct 30 17:45:03.197967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.196277    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerDied","Data":"42368e17030ac26d1d0e9f445c2fa53229ec662447f43c49fffdf32fb1cc8e9a"}
Oct 30 17:45:03.197967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.197042    5534 scope.go:122] "RemoveContainer" containerID="42368e17030ac26d1d0e9f445c2fa53229ec662447f43c49fffdf32fb1cc8e9a"
Oct 30 17:45:03.201902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.201874    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerStarted","Data":"481832249bececbcfe2ce3e85c9954df5e32096b0d4565d8502e888c374cb43d"}
Oct 30 17:45:03.250014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.235024    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerStarted","Data":"7c71f364b28da9a1bcf59c9336522a5e2a105c8ab206b451eac3c6cea7a0db4b"}
Oct 30 17:45:03.250014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.236084    5534 scope.go:122] "RemoveContainer" containerID="7c71f364b28da9a1bcf59c9336522a5e2a105c8ab206b451eac3c6cea7a0db4b"
Oct 30 17:45:03.259723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.252798    5534 generic.go:356] "Generic (PLEG): container finished" podID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerID="dca2f091e01197ab80b7082985698ef8159248bd1853ce147690bc4c6ab5c44e" exitCode=60
Oct 30 17:45:03.259723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.252895    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerDied","Data":"dca2f091e01197ab80b7082985698ef8159248bd1853ce147690bc4c6ab5c44e"}
Oct 30 17:45:03.290567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.288251    5534 scope.go:122] "RemoveContainer" containerID="dca2f091e01197ab80b7082985698ef8159248bd1853ce147690bc4c6ab5c44e"
Oct 30 17:45:03.306252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.305884    5534 generic.go:356] "Generic (PLEG): container finished" podID="f2a63f21-1afe-421e-9b25-91c082d51442" containerID="37e96cd347e94a29bd1718b464ebd950ec7fc46bf1ef033d8ebdf17ab9e4aff6" exitCode=60
Oct 30 17:45:03.306252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.305953    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" event={"ID":"f2a63f21-1afe-421e-9b25-91c082d51442","Type":"ContainerDied","Data":"37e96cd347e94a29bd1718b464ebd950ec7fc46bf1ef033d8ebdf17ab9e4aff6"}
Oct 30 17:45:03.306391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.306316    5534 scope.go:122] "RemoveContainer" containerID="37e96cd347e94a29bd1718b464ebd950ec7fc46bf1ef033d8ebdf17ab9e4aff6"
Oct 30 17:45:03.308401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.308378    5534 generic.go:356] "Generic (PLEG): container finished" podID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerID="22f7670a8e85641495065ee7bddff2831148c77a60a780be54abd667808fe626" exitCode=60
Oct 30 17:45:03.308497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.308456    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerDied","Data":"22f7670a8e85641495065ee7bddff2831148c77a60a780be54abd667808fe626"}
Oct 30 17:45:03.308881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.308865    5534 scope.go:122] "RemoveContainer" containerID="22f7670a8e85641495065ee7bddff2831148c77a60a780be54abd667808fe626"
Oct 30 17:45:03.310683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.310627    5534 generic.go:356] "Generic (PLEG): container finished" podID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerID="c015e51f0bb1a8514192441c61554a8e4f1aab5337e36126d5388e44055f442a" exitCode=60
Oct 30 17:45:03.310841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.310693    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerDied","Data":"c015e51f0bb1a8514192441c61554a8e4f1aab5337e36126d5388e44055f442a"}
Oct 30 17:45:03.311126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.311106    5534 scope.go:122] "RemoveContainer" containerID="c015e51f0bb1a8514192441c61554a8e4f1aab5337e36126d5388e44055f442a"
Oct 30 17:45:03.313564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.313544    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" event={"ID":"105ab0a0-4585-4c94-8eb6-c789cd5b00a1","Type":"ContainerStarted","Data":"df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b"}
Oct 30 17:45:03.317549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.317523    5534 generic.go:356] "Generic (PLEG): container finished" podID="83712796-730a-494d-9fd3-805217d6823e" containerID="6c0cdffda10ddf3ec14375d42136385142ac9aed1d67286992ba6abaea7585f9" exitCode=60
Oct 30 17:45:03.317657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.317619    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerDied","Data":"6c0cdffda10ddf3ec14375d42136385142ac9aed1d67286992ba6abaea7585f9"}
Oct 30 17:45:03.317996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.317975    5534 scope.go:122] "RemoveContainer" containerID="6c0cdffda10ddf3ec14375d42136385142ac9aed1d67286992ba6abaea7585f9"
Oct 30 17:45:03.318754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.318717    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" podStartSLOduration=3.318704623 podStartE2EDuration="3.318704623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:03.314126081 +0000 UTC m=+1909.076169896" watchObservedRunningTime="2026-10-30 17:45:03.318704623 +0000 UTC m=+1909.080748436"
Oct 30 17:45:03.321020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.320999    5534 generic.go:356] "Generic (PLEG): container finished" podID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerID="0ba7c4535104e061f0b9faffdebed12faf0b405ddc0fb3768f57d1a04887a5de" exitCode=60
Oct 30 17:45:03.321112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.321078    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerDied","Data":"0ba7c4535104e061f0b9faffdebed12faf0b405ddc0fb3768f57d1a04887a5de"}
Oct 30 17:45:03.321344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.321300    5534 scope.go:122] "RemoveContainer" containerID="0ba7c4535104e061f0b9faffdebed12faf0b405ddc0fb3768f57d1a04887a5de"
Oct 30 17:45:03.322684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.321830    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:45:03.325016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.324993    5534 generic.go:356] "Generic (PLEG): container finished" podID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerID="3556836c57d364a5c388315398bdf28075246766688a9712f822269978cd0f7b" exitCode=60
Oct 30 17:45:03.325110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.325052    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" event={"ID":"85827e4b-8f28-4b11-a1bb-b8b78dd94d43","Type":"ContainerDied","Data":"3556836c57d364a5c388315398bdf28075246766688a9712f822269978cd0f7b"}
Oct 30 17:45:03.325260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.325247    5534 scope.go:122] "RemoveContainer" containerID="3556836c57d364a5c388315398bdf28075246766688a9712f822269978cd0f7b"
Oct 30 17:45:03.325878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.325855    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:03.422684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.422063    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd\" (UniqueName: \"kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd\") pod \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") "
Oct 30 17:45:03.422684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.422243    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume\") pod \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\" (UID: \"78ba9c34-8c81-4b68-90dc-baa751b5cb93\") "
Oct 30 17:45:03.425799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.424929    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume" pod "78ba9c34-8c81-4b68-90dc-baa751b5cb93" (UID: "78ba9c34-8c81-4b68-90dc-baa751b5cb93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:03.489084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.488052    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd" pod "78ba9c34-8c81-4b68-90dc-baa751b5cb93" (UID: "78ba9c34-8c81-4b68-90dc-baa751b5cb93"). InnerVolumeSpecName "kube-api-access-6ddsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:03.524856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.524824    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ddsd\" (UniqueName: \"kubernetes.io/projected/78ba9c34-8c81-4b68-90dc-baa751b5cb93-kube-api-access-6ddsd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:03.524856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:03.524853    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ba9c34-8c81-4b68-90dc-baa751b5cb93-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:04.143829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.143794    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:45:04.144308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.144068    5534 patch_prober.go:29] interesting pod/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 17:45:04.144308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.144126    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 17:45:04.336227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.336189    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerStarted","Data":"6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906"}
Oct 30 17:45:04.338030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.337996    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" event={"ID":"6f15f5e4-aa84-478e-9111-bff23ac5463a","Type":"ContainerStarted","Data":"fb9f201fc8eac98b2c2706fe3f6cb806c00cdd7b439de9d6c60bca51f25469f7"}
Oct 30 17:45:04.340040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.339911    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerStarted","Data":"877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598"}
Oct 30 17:45:04.341428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.341403    5534 generic.go:356] "Generic (PLEG): container finished" podID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerID="481832249bececbcfe2ce3e85c9954df5e32096b0d4565d8502e888c374cb43d" exitCode=60
Oct 30 17:45:04.341519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.341489    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerDied","Data":"481832249bececbcfe2ce3e85c9954df5e32096b0d4565d8502e888c374cb43d"}
Oct 30 17:45:04.341654 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.341624    5534 scope.go:122] "RemoveContainer" containerID="481832249bececbcfe2ce3e85c9954df5e32096b0d4565d8502e888c374cb43d"
Oct 30 17:45:04.343364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.343339    5534 generic.go:356] "Generic (PLEG): container finished" podID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerID="7c71f364b28da9a1bcf59c9336522a5e2a105c8ab206b451eac3c6cea7a0db4b" exitCode=60
Oct 30 17:45:04.343457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.343444    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerDied","Data":"7c71f364b28da9a1bcf59c9336522a5e2a105c8ab206b451eac3c6cea7a0db4b"}
Oct 30 17:45:04.343497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.343476    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerStarted","Data":"c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e"}
Oct 30 17:45:04.348038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.348017    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerStarted","Data":"22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8"}
Oct 30 17:45:04.350110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.350087    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" event={"ID":"f2a63f21-1afe-421e-9b25-91c082d51442","Type":"ContainerStarted","Data":"8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521"}
Oct 30 17:45:04.352036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.352014    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerStarted","Data":"d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1"}
Oct 30 17:45:04.397730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.390458    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" podStartSLOduration=4.390437109 podStartE2EDuration="4.390437109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:04.377352155 +0000 UTC m=+1910.139395969" watchObservedRunningTime="2026-10-30 17:45:04.390437109 +0000 UTC m=+1910.152480926"
Oct 30 17:45:04.412205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.412162    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerStarted","Data":"ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378"}
Oct 30 17:45:04.412390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.412211    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerStarted","Data":"a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333"}
Oct 30 17:45:04.412539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.412517    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerStarted","Data":"74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee"}
Oct 30 17:45:04.415924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.415875    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" event={"ID":"85827e4b-8f28-4b11-a1bb-b8b78dd94d43","Type":"ContainerStarted","Data":"2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e"}
Oct 30 17:45:04.416018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.415959    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"
Oct 30 17:45:04.628717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.628633    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" podStartSLOduration=4.628613362 podStartE2EDuration="4.628613362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:04.628084615 +0000 UTC m=+1910.390128429" watchObservedRunningTime="2026-10-30 17:45:04.628613362 +0000 UTC m=+1910.390657174"
Oct 30 17:45:04.743037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.742976    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" podStartSLOduration=4.742958009 podStartE2EDuration="4.742958009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:04.700829576 +0000 UTC m=+1910.462873389" watchObservedRunningTime="2026-10-30 17:45:04.742958009 +0000 UTC m=+1910.505001821"
Oct 30 17:45:04.743240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.743081    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" podStartSLOduration=4.743074063 podStartE2EDuration="4.743074063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:04.741866677 +0000 UTC m=+1910.503910500" watchObservedRunningTime="2026-10-30 17:45:04.743074063 +0000 UTC m=+1910.505117879"
Oct 30 17:45:04.908335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.908259    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" podStartSLOduration=4.908236238 podStartE2EDuration="4.908236238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:04.860907195 +0000 UTC m=+1910.622951009" watchObservedRunningTime="2026-10-30 17:45:04.908236238 +0000 UTC m=+1910.670280053"
Oct 30 17:45:04.914398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.913867    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"]
Oct 30 17:45:04.918596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.918548    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-74762d-6594787f9b-4dxl4"]
Oct 30 17:45:04.981071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:04.980994    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" podStartSLOduration=4.980974718 podStartE2EDuration="4.980974718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:04.980964294 +0000 UTC m=+1910.743008114" watchObservedRunningTime="2026-10-30 17:45:04.980974718 +0000 UTC m=+1910.743018532"
Oct 30 17:45:05.027343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.027286    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" podStartSLOduration=5.027270893 podStartE2EDuration="5.027270893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:05.026693906 +0000 UTC m=+1910.788737720" watchObservedRunningTime="2026-10-30 17:45:05.027270893 +0000 UTC m=+1910.789314705"
Oct 30 17:45:05.144296 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.144262    5534 patch_prober.go:29] interesting pod/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 17:45:05.144704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.144317    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 17:45:05.427761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.427725    5534 generic.go:356] "Generic (PLEG): container finished" podID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerID="c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e" exitCode=60
Oct 30 17:45:05.427932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.427827    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerDied","Data":"c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e"}
Oct 30 17:45:05.427932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.427901    5534 scope.go:122] "RemoveContainer" containerID="7c71f364b28da9a1bcf59c9336522a5e2a105c8ab206b451eac3c6cea7a0db4b"
Oct 30 17:45:05.428221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.428203    5534 scope.go:122] "RemoveContainer" containerID="c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e"
Oct 30 17:45:05.428679 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.428620    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4_odh-ai-gateway-infra(aac47d37-9668-42b9-a8c7-e24eb5d1afa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9"
Oct 30 17:45:05.430504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.430481    5534 generic.go:356] "Generic (PLEG): container finished" podID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerID="22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8" exitCode=60
Oct 30 17:45:05.430635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.430568    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerDied","Data":"22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8"}
Oct 30 17:45:05.431000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.430980    5534 scope.go:122] "RemoveContainer" containerID="22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8"
Oct 30 17:45:05.431426 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.431402    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs_odh-ai-gateway-infra(9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a"
Oct 30 17:45:05.433095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.433070    5534 generic.go:356] "Generic (PLEG): container finished" podID="f2a63f21-1afe-421e-9b25-91c082d51442" containerID="8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521" exitCode=60
Oct 30 17:45:05.433223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.433163    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" event={"ID":"f2a63f21-1afe-421e-9b25-91c082d51442","Type":"ContainerDied","Data":"8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521"}
Oct 30 17:45:05.433468 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.433449    5534 scope.go:122] "RemoveContainer" containerID="8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521"
Oct 30 17:45:05.433911 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.433892    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889705-kk7st_odh-ai-gateway-infra(f2a63f21-1afe-421e-9b25-91c082d51442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" podUID="f2a63f21-1afe-421e-9b25-91c082d51442"
Oct 30 17:45:05.435658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.435590    5534 generic.go:356] "Generic (PLEG): container finished" podID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerID="d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1" exitCode=60
Oct 30 17:45:05.435658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.435677    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerDied","Data":"d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1"}
Oct 30 17:45:05.436045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.436029    5534 scope.go:122] "RemoveContainer" containerID="d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1"
Oct 30 17:45:05.436466 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.436416    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49_odh-ai-gateway-infra(05d7ba6a-9ead-4270-a755-2ccbec88707e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e"
Oct 30 17:45:05.437932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.437911    5534 generic.go:356] "Generic (PLEG): container finished" podID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerID="ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378" exitCode=60
Oct 30 17:45:05.438017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.437993    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerDied","Data":"ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378"}
Oct 30 17:45:05.438277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.438262    5534 scope.go:122] "RemoveContainer" containerID="ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378"
Oct 30 17:45:05.438677 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.438630    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8_odh-ai-gateway-infra(6d3b6fe8-94ca-444d-90ef-78aa4bbccf58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58"
Oct 30 17:45:05.440227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.440206    5534 generic.go:356] "Generic (PLEG): container finished" podID="83712796-730a-494d-9fd3-805217d6823e" containerID="a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333" exitCode=60
Oct 30 17:45:05.440313 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.440290    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerDied","Data":"a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333"}
Oct 30 17:45:05.440578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.440560    5534 scope.go:122] "RemoveContainer" containerID="a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333"
Oct 30 17:45:05.441321 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.440988    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6_odh-ai-gateway-infra(83712796-730a-494d-9fd3-805217d6823e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" podUID="83712796-730a-494d-9fd3-805217d6823e"
Oct 30 17:45:05.442549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.442523    5534 generic.go:356] "Generic (PLEG): container finished" podID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerID="74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee" exitCode=60
Oct 30 17:45:05.442706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.442591    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerDied","Data":"74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee"}
Oct 30 17:45:05.442970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.442953    5534 scope.go:122] "RemoveContainer" containerID="74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee"
Oct 30 17:45:05.443289 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.443273    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq_odh-ai-gateway-infra(d7a5d82b-7418-42a6-8265-1a889f3ebecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf"
Oct 30 17:45:05.444600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.444581    5534 scope.go:122] "RemoveContainer" containerID="dca2f091e01197ab80b7082985698ef8159248bd1853ce147690bc4c6ab5c44e"
Oct 30 17:45:05.445145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.445124    5534 generic.go:356] "Generic (PLEG): container finished" podID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerID="2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e" exitCode=60
Oct 30 17:45:05.445240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.445218    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" event={"ID":"85827e4b-8f28-4b11-a1bb-b8b78dd94d43","Type":"ContainerDied","Data":"2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e"}
Oct 30 17:45:05.445486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.445464    5534 scope.go:122] "RemoveContainer" containerID="2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e"
Oct 30 17:45:05.446139 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.446116    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c_odh-ai-gateway-infra(85827e4b-8f28-4b11-a1bb-b8b78dd94d43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43"
Oct 30 17:45:05.447410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.447389    5534 generic.go:356] "Generic (PLEG): container finished" podID="95e9bdd1-ed4d-493a-b112-e24721540858" containerID="6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906" exitCode=60
Oct 30 17:45:05.447533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.447471    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerDied","Data":"6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906"}
Oct 30 17:45:05.447746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.447733    5534 scope.go:122] "RemoveContainer" containerID="6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906"
Oct 30 17:45:05.448061 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.448039    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k_odh-ai-gateway-infra(95e9bdd1-ed4d-493a-b112-e24721540858)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" podUID="95e9bdd1-ed4d-493a-b112-e24721540858"
Oct 30 17:45:05.449805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.449784    5534 generic.go:356] "Generic (PLEG): container finished" podID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerID="fb9f201fc8eac98b2c2706fe3f6cb806c00cdd7b439de9d6c60bca51f25469f7" exitCode=60
Oct 30 17:45:05.449909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.449878    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" event={"ID":"6f15f5e4-aa84-478e-9111-bff23ac5463a","Type":"ContainerDied","Data":"fb9f201fc8eac98b2c2706fe3f6cb806c00cdd7b439de9d6c60bca51f25469f7"}
Oct 30 17:45:05.450232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.450166    5534 scope.go:122] "RemoveContainer" containerID="fb9f201fc8eac98b2c2706fe3f6cb806c00cdd7b439de9d6c60bca51f25469f7"
Oct 30 17:45:05.450772 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.450749    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk_odh-ai-gateway-infra(6f15f5e4-aa84-478e-9111-bff23ac5463a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a"
Oct 30 17:45:05.452591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.452565    5534 generic.go:356] "Generic (PLEG): container finished" podID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerID="877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598" exitCode=60
Oct 30 17:45:05.452767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.452737    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerDied","Data":"877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598"}
Oct 30 17:45:05.463673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.457477    5534 scope.go:122] "RemoveContainer" containerID="877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598"
Oct 30 17:45:05.463673 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:05.458006    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j_odh-ai-gateway-infra(3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d"
Oct 30 17:45:05.500624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.500579    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerStarted","Data":"25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d"}
Oct 30 17:45:05.502025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.501973    5534 scope.go:122] "RemoveContainer" containerID="37e96cd347e94a29bd1718b464ebd950ec7fc46bf1ef033d8ebdf17ab9e4aff6"
Oct 30 17:45:05.521185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.521160    5534 scope.go:122] "RemoveContainer" containerID="22f7670a8e85641495065ee7bddff2831148c77a60a780be54abd667808fe626"
Oct 30 17:45:05.539398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.539374    5534 scope.go:122] "RemoveContainer" containerID="c015e51f0bb1a8514192441c61554a8e4f1aab5337e36126d5388e44055f442a"
Oct 30 17:45:05.561705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.552869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:05.613268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.612901    5534 scope.go:122] "RemoveContainer" containerID="6c0cdffda10ddf3ec14375d42136385142ac9aed1d67286992ba6abaea7585f9"
Oct 30 17:45:05.648060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.648007    5534 scope.go:122] "RemoveContainer" containerID="0ba7c4535104e061f0b9faffdebed12faf0b405ddc0fb3768f57d1a04887a5de"
Oct 30 17:45:05.771049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.770414    5534 scope.go:122] "RemoveContainer" containerID="3556836c57d364a5c388315398bdf28075246766688a9712f822269978cd0f7b"
Oct 30 17:45:05.804614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.804564    5534 scope.go:122] "RemoveContainer" containerID="9cf2377fab03bd8752a9e0f828c62b0f1652d6b74fc8f631c6d5b3fe32f449a1"
Oct 30 17:45:05.822893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.822469    5534 scope.go:122] "RemoveContainer" containerID="fbadda332c790a221de7264292553be40b88cbcd4ccca714ef69d7b0b2144f3e"
Oct 30 17:45:05.840857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:05.840824    5534 scope.go:122] "RemoveContainer" containerID="42368e17030ac26d1d0e9f445c2fa53229ec662447f43c49fffdf32fb1cc8e9a"
Oct 30 17:45:06.143954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.143736    5534 patch_prober.go:29] interesting pod/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 17:45:06.144466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.144432    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 17:45:06.400205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.400111    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ba9c34-8c81-4b68-90dc-baa751b5cb93" path="/var/lib/kubelet/pods/78ba9c34-8c81-4b68-90dc-baa751b5cb93/volumes"
Oct 30 17:45:06.512364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.512336    5534 scope.go:122] "RemoveContainer" containerID="8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521"
Oct 30 17:45:06.513031 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.513006    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889705-kk7st_odh-ai-gateway-infra(f2a63f21-1afe-421e-9b25-91c082d51442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" podUID="f2a63f21-1afe-421e-9b25-91c082d51442"
Oct 30 17:45:06.514554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.514532    5534 scope.go:122] "RemoveContainer" containerID="d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1"
Oct 30 17:45:06.515103 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.515074    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49_odh-ai-gateway-infra(05d7ba6a-9ead-4270-a755-2ccbec88707e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e"
Oct 30 17:45:06.516760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.516736    5534 scope.go:122] "RemoveContainer" containerID="ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378"
Oct 30 17:45:06.517162 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.517135    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8_odh-ai-gateway-infra(6d3b6fe8-94ca-444d-90ef-78aa4bbccf58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58"
Oct 30 17:45:06.518718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.518630    5534 scope.go:122] "RemoveContainer" containerID="a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333"
Oct 30 17:45:06.519024 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.519004    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6_odh-ai-gateway-infra(83712796-730a-494d-9fd3-805217d6823e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" podUID="83712796-730a-494d-9fd3-805217d6823e"
Oct 30 17:45:06.520454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.520434    5534 scope.go:122] "RemoveContainer" containerID="74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee"
Oct 30 17:45:06.520807 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.520780    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq_odh-ai-gateway-infra(d7a5d82b-7418-42a6-8265-1a889f3ebecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf"
Oct 30 17:45:06.522533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.522514    5534 scope.go:122] "RemoveContainer" containerID="2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e"
Oct 30 17:45:06.522939 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.522911    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c_odh-ai-gateway-infra(85827e4b-8f28-4b11-a1bb-b8b78dd94d43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43"
Oct 30 17:45:06.524385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.524367    5534 scope.go:122] "RemoveContainer" containerID="6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906"
Oct 30 17:45:06.524766 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.524729    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k_odh-ai-gateway-infra(95e9bdd1-ed4d-493a-b112-e24721540858)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" podUID="95e9bdd1-ed4d-493a-b112-e24721540858"
Oct 30 17:45:06.526421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.526402    5534 scope.go:122] "RemoveContainer" containerID="fb9f201fc8eac98b2c2706fe3f6cb806c00cdd7b439de9d6c60bca51f25469f7"
Oct 30 17:45:06.526721 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.526700    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk_odh-ai-gateway-infra(6f15f5e4-aa84-478e-9111-bff23ac5463a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a"
Oct 30 17:45:06.528627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.528610    5534 scope.go:122] "RemoveContainer" containerID="877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598"
Oct 30 17:45:06.528935 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.528907    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j_odh-ai-gateway-infra(3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d"
Oct 30 17:45:06.530781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.530754    5534 generic.go:356] "Generic (PLEG): container finished" podID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerID="25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d" exitCode=60
Oct 30 17:45:06.530890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.530797    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerDied","Data":"25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d"}
Oct 30 17:45:06.530890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.530860    5534 scope.go:122] "RemoveContainer" containerID="481832249bececbcfe2ce3e85c9954df5e32096b0d4565d8502e888c374cb43d"
Oct 30 17:45:06.531227 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.531184    5534 scope.go:122] "RemoveContainer" containerID="25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d"
Oct 30 17:45:06.531541 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.531516    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt_odh-ai-gateway-infra(2e1f04ed-4c86-4c75-bdc2-20aa12564d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40"
Oct 30 17:45:06.533322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.533304    5534 scope.go:122] "RemoveContainer" containerID="c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e"
Oct 30 17:45:06.533691 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.533623    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4_odh-ai-gateway-infra(aac47d37-9668-42b9-a8c7-e24eb5d1afa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9"
Oct 30 17:45:06.535032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:06.535013    5534 scope.go:122] "RemoveContainer" containerID="22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8"
Oct 30 17:45:06.535307 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:06.535286    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs_odh-ai-gateway-infra(9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a"
Oct 30 17:45:07.145135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:07.145093    5534 patch_prober.go:29] interesting pod/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 17:45:07.145769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:07.145730    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 17:45:07.491159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:07.491053    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:07.549223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:07.549193    5534 scope.go:122] "RemoveContainer" containerID="25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d"
Oct 30 17:45:07.549603 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:07.549581    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt_odh-ai-gateway-infra(2e1f04ed-4c86-4c75-bdc2-20aa12564d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40"
Oct 30 17:45:08.143328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:08.143292    5534 patch_prober.go:29] interesting pod/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 17:45:08.143497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:08.143344    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 17:45:09.147955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:09.147921    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:45:09.148508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:09.148489    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:45:09.149414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:09.149393    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:45:09.310696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:09.309624    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:09.429174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:09.429078    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:11.200784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:11.200736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:12.927221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:12.927182    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:14.472287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:14.472244    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"]
Oct 30 17:45:14.486074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:14.486037    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:14.868666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:14.863063    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:15.002003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.001964    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca\") pod \"6f15f5e4-aa84-478e-9111-bff23ac5463a\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") "
Oct 30 17:45:15.002234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.002215    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll\" (UniqueName: \"kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll\") pod \"6f15f5e4-aa84-478e-9111-bff23ac5463a\" (UID: \"6f15f5e4-aa84-478e-9111-bff23ac5463a\") "
Oct 30 17:45:15.005938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.005903    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll" pod "6f15f5e4-aa84-478e-9111-bff23ac5463a" (UID: "6f15f5e4-aa84-478e-9111-bff23ac5463a"). InnerVolumeSpecName "kube-api-access-pbfll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:15.006171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.006144    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca" pod "6f15f5e4-aa84-478e-9111-bff23ac5463a" (UID: "6f15f5e4-aa84-478e-9111-bff23ac5463a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:15.120768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.120681    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f15f5e4-aa84-478e-9111-bff23ac5463a-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:15.120976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.120960    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbfll\" (UniqueName: \"kubernetes.io/projected/6f15f5e4-aa84-478e-9111-bff23ac5463a-kube-api-access-pbfll\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:15.175939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.175859    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:15.263232 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:15.263138    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:45:15.362631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.362600    5534 scope.go:122] "RemoveContainer" containerID="877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598"
Oct 30 17:45:15.362829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.362809    5534 scope.go:122] "RemoveContainer" containerID="8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521"
Oct 30 17:45:15.362999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.362977    5534 scope.go:122] "RemoveContainer" containerID="2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e"
Oct 30 17:45:15.496790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.496749    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:15.529100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.529067    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"]
Oct 30 17:45:15.549587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.549550    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cc4e96-k7v74"]
Oct 30 17:45:15.626278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.626240    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:15.647347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.647258    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"
Oct 30 17:45:15.647347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.647270    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk" event={"ID":"6f15f5e4-aa84-478e-9111-bff23ac5463a","Type":"ContainerDied","Data":"5d36bc8084a69972ba73818d7614275d0725d676582eccb1ae21371687810e41"}
Oct 30 17:45:15.647347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.647328    5534 scope.go:122] "RemoveContainer" containerID="fb9f201fc8eac98b2c2706fe3f6cb806c00cdd7b439de9d6c60bca51f25469f7"
Oct 30 17:45:15.725073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.725032    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"]
Oct 30 17:45:15.729305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:15.729272    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cc4e96-29889705-z2qvk"]
Oct 30 17:45:16.099767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.099728    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:16.362559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.362528    5534 scope.go:122] "RemoveContainer" containerID="25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d"
Oct 30 17:45:16.377275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.377243    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" path="/var/lib/kubelet/pods/6f15f5e4-aa84-478e-9111-bff23ac5463a/volumes"
Oct 30 17:45:16.378043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.378022    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7a50f1-96c7-4639-a707-56f5bfef08b3" path="/var/lib/kubelet/pods/de7a50f1-96c7-4639-a707-56f5bfef08b3/volumes"
Oct 30 17:45:16.502092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.502053    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:16.661710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.661622    5534 generic.go:356] "Generic (PLEG): container finished" podID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerID="d73f6f0836fbde3b197dd38b32bd2b4e8aef5cb87bb50e1401ecdfdfa0a9424b" exitCode=60
Oct 30 17:45:16.661899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.661833    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" event={"ID":"85827e4b-8f28-4b11-a1bb-b8b78dd94d43","Type":"ContainerDied","Data":"d73f6f0836fbde3b197dd38b32bd2b4e8aef5cb87bb50e1401ecdfdfa0a9424b"}
Oct 30 17:45:16.661995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.661898    5534 scope.go:122] "RemoveContainer" containerID="2205dd93149ea6ee6eee5bc18e22bb8a6c0b74c4bf091a15af0792f49737881e"
Oct 30 17:45:16.662478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.662458    5534 scope.go:122] "RemoveContainer" containerID="d73f6f0836fbde3b197dd38b32bd2b4e8aef5cb87bb50e1401ecdfdfa0a9424b"
Oct 30 17:45:16.662968 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:16.662933    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c_odh-ai-gateway-infra(85827e4b-8f28-4b11-a1bb-b8b78dd94d43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43"
Oct 30 17:45:16.669658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.669604    5534 generic.go:356] "Generic (PLEG): container finished" podID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerID="00cb2cff0c71fd13bb3f9335b1da4bd4214e954047b3feae300f624d66a39d65" exitCode=60
Oct 30 17:45:16.669844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.669827    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerDied","Data":"00cb2cff0c71fd13bb3f9335b1da4bd4214e954047b3feae300f624d66a39d65"}
Oct 30 17:45:16.670448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.670421    5534 scope.go:122] "RemoveContainer" containerID="00cb2cff0c71fd13bb3f9335b1da4bd4214e954047b3feae300f624d66a39d65"
Oct 30 17:45:16.670936 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:16.670905    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j_odh-ai-gateway-infra(3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d"
Oct 30 17:45:16.674263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.674229    5534 generic.go:356] "Generic (PLEG): container finished" podID="f2a63f21-1afe-421e-9b25-91c082d51442" containerID="ece2826db5445060a48de3c22b8bb89587b523df760ee4a29a9df07b8175196d" exitCode=60
Oct 30 17:45:16.674403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.674384    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" event={"ID":"f2a63f21-1afe-421e-9b25-91c082d51442","Type":"ContainerDied","Data":"ece2826db5445060a48de3c22b8bb89587b523df760ee4a29a9df07b8175196d"}
Oct 30 17:45:16.675018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.674996    5534 scope.go:122] "RemoveContainer" containerID="ece2826db5445060a48de3c22b8bb89587b523df760ee4a29a9df07b8175196d"
Oct 30 17:45:16.675584 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:16.675521    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889705-kk7st_odh-ai-gateway-infra(f2a63f21-1afe-421e-9b25-91c082d51442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" podUID="f2a63f21-1afe-421e-9b25-91c082d51442"
Oct 30 17:45:16.727407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.727384    5534 scope.go:122] "RemoveContainer" containerID="877b1238e0fb126c36c07988dfe94ad581d6d0895ed50aaf7d54b54563a51598"
Oct 30 17:45:16.743090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:16.743069    5534 scope.go:122] "RemoveContainer" containerID="8bc7a8f661145654e10d10f9048d5a1307abae14587d9ddd1c40c4236dfb9521"
Oct 30 17:45:17.362590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.362563    5534 scope.go:122] "RemoveContainer" containerID="a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333"
Oct 30 17:45:17.407569 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.407534    5534 scope.go:122] "RemoveContainer" containerID="7355bf2fb14cc3a0a808e5db49721ae58c0ba6e9fbe62476a379a592166b5f35"
Oct 30 17:45:17.443396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.443372    5534 scope.go:122] "RemoveContainer" containerID="7a3afdf05cb3adbcca4be479fd00bc115a41597929fa18e33b5c0c896b3160ef"
Oct 30 17:45:17.458776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.458739    5534 scope.go:122] "RemoveContainer" containerID="eb60b4a749f60d6d6e688d1f9585fd23a175cdd327f56eaf474d8698fd181392"
Oct 30 17:45:17.475438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.475376    5534 scope.go:122] "RemoveContainer" containerID="79955b7c1e6b3f68b7e7523c761a8979ce32e51bc6847f9fcbde00b8982299fd"
Oct 30 17:45:17.568192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.568166    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:17.706319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.706231    5534 generic.go:356] "Generic (PLEG): container finished" podID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerID="182602903d2041b207e450a28bb7a403bab50e765ef2bed50d76df847761d45c" exitCode=60
Oct 30 17:45:17.706319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.706280    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerDied","Data":"182602903d2041b207e450a28bb7a403bab50e765ef2bed50d76df847761d45c"}
Oct 30 17:45:17.706515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.706336    5534 scope.go:122] "RemoveContainer" containerID="25484a69f402e6ed1cc80d879d0436fc4cfe8debf7d9b11b36466f3c8c201c7d"
Oct 30 17:45:17.706944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.706918    5534 scope.go:122] "RemoveContainer" containerID="182602903d2041b207e450a28bb7a403bab50e765ef2bed50d76df847761d45c"
Oct 30 17:45:17.707782 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:17.707368    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt_odh-ai-gateway-infra(2e1f04ed-4c86-4c75-bdc2-20aa12564d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40"
Oct 30 17:45:17.716543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.716511    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"]
Oct 30 17:45:17.761205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.761165    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"]
Oct 30 17:45:17.821102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:17.809743    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"]
Oct 30 17:45:18.136687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.136034    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:18.142719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.137328    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:18.179294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.178801    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca\") pod \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") "
Oct 30 17:45:18.179294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.179074    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv\" (UniqueName: \"kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv\") pod \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\" (UID: \"85827e4b-8f28-4b11-a1bb-b8b78dd94d43\") "
Oct 30 17:45:18.185564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.181046    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca" pod "85827e4b-8f28-4b11-a1bb-b8b78dd94d43" (UID: "85827e4b-8f28-4b11-a1bb-b8b78dd94d43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:18.211143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.205457    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv" pod "85827e4b-8f28-4b11-a1bb-b8b78dd94d43" (UID: "85827e4b-8f28-4b11-a1bb-b8b78dd94d43"). InnerVolumeSpecName "kube-api-access-lg5tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:18.258292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.249014    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:18.281921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.280578    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc\" (UniqueName: \"kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc\") pod \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") "
Oct 30 17:45:18.281921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.280954    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca\") pod \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\" (UID: \"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d\") "
Oct 30 17:45:18.281921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.281621    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:18.281921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.281664    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5tv\" (UniqueName: \"kubernetes.io/projected/85827e4b-8f28-4b11-a1bb-b8b78dd94d43-kube-api-access-lg5tv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:18.282833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.282491    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca" pod "3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" (UID: "3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:18.291047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.290991    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc" pod "3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" (UID: "3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d"). InnerVolumeSpecName "kube-api-access-stkvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:18.393679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.392701    5534 scope.go:122] "RemoveContainer" containerID="c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e"
Oct 30 17:45:18.393679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.393135    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:18.393679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.393140    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:18.393679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.393175    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stkvc\" (UniqueName: \"kubernetes.io/projected/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d-kube-api-access-stkvc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:18.393679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.393595    5534 scope.go:122] "RemoveContainer" containerID="d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1"
Oct 30 17:45:18.394460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.394353    5534 scope.go:122] "RemoveContainer" containerID="74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee"
Oct 30 17:45:18.395012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.394955    5534 scope.go:122] "RemoveContainer" containerID="6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906"
Oct 30 17:45:18.404046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.395520    5534 scope.go:122] "RemoveContainer" containerID="ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378"
Oct 30 17:45:18.493993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.493958    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7\" (UniqueName: \"kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7\") pod \"f2a63f21-1afe-421e-9b25-91c082d51442\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") "
Oct 30 17:45:18.494126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.494111    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca\") pod \"f2a63f21-1afe-421e-9b25-91c082d51442\" (UID: \"f2a63f21-1afe-421e-9b25-91c082d51442\") "
Oct 30 17:45:18.507702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.497930    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca" pod "f2a63f21-1afe-421e-9b25-91c082d51442" (UID: "f2a63f21-1afe-421e-9b25-91c082d51442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:18.520514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.520457    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7" pod "f2a63f21-1afe-421e-9b25-91c082d51442" (UID: "f2a63f21-1afe-421e-9b25-91c082d51442"). InnerVolumeSpecName "kube-api-access-hndx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:18.596580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.596544    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a63f21-1afe-421e-9b25-91c082d51442-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:18.596580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.596576    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndx7\" (UniqueName: \"kubernetes.io/projected/f2a63f21-1afe-421e-9b25-91c082d51442-kube-api-access-hndx7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:18.717505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.717466    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"
Oct 30 17:45:18.717726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.717466    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st" event={"ID":"f2a63f21-1afe-421e-9b25-91c082d51442","Type":"ContainerDied","Data":"17af438a12b1184aafebc5835c6ae1ecf172ffb43c7cd0f308018755eef18097"}
Oct 30 17:45:18.717841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.717794    5534 scope.go:122] "RemoveContainer" containerID="ece2826db5445060a48de3c22b8bb89587b523df760ee4a29a9df07b8175196d"
Oct 30 17:45:18.720413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.720380    5534 generic.go:356] "Generic (PLEG): container finished" podID="83712796-730a-494d-9fd3-805217d6823e" containerID="6feee743922c4dda803d7ed4dd0e2a09422ca7e442f3a6181d47cce3df5d6704" exitCode=60
Oct 30 17:45:18.720575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.720416    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerDied","Data":"6feee743922c4dda803d7ed4dd0e2a09422ca7e442f3a6181d47cce3df5d6704"}
Oct 30 17:45:18.720971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.720954    5534 scope.go:122] "RemoveContainer" containerID="6feee743922c4dda803d7ed4dd0e2a09422ca7e442f3a6181d47cce3df5d6704"
Oct 30 17:45:18.721394 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:18.721367    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6_odh-ai-gateway-infra(83712796-730a-494d-9fd3-805217d6823e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" podUID="83712796-730a-494d-9fd3-805217d6823e"
Oct 30 17:45:18.722327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.722310    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"
Oct 30 17:45:18.722424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.722330    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c" event={"ID":"85827e4b-8f28-4b11-a1bb-b8b78dd94d43","Type":"ContainerDied","Data":"22f3974b33015609add8060cf1c576efce1110ac43c4409256044359c2f208be"}
Oct 30 17:45:18.724052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.724028    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j" event={"ID":"3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d","Type":"ContainerDied","Data":"4caf2df334756fe6b2f0d1489b8a4924d9654e8d8b07a3521afaf94cc4ad2aa3"}
Oct 30 17:45:18.724146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.724129    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"
Oct 30 17:45:18.766269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.765501    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"]
Oct 30 17:45:18.767703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.767564    5534 scope.go:122] "RemoveContainer" containerID="a8b281c99058f525bd08e96833220594407f92bb1e81659f0908a8ebf95bb333"
Oct 30 17:45:18.776582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.776356    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"]
Oct 30 17:45:18.792075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.792038    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kk7st"]
Oct 30 17:45:18.792611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.792585    5534 scope.go:122] "RemoveContainer" containerID="d73f6f0836fbde3b197dd38b32bd2b4e8aef5cb87bb50e1401ecdfdfa0a9424b"
Oct 30 17:45:18.811428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.811313    5534 scope.go:122] "RemoveContainer" containerID="00cb2cff0c71fd13bb3f9335b1da4bd4214e954047b3feae300f624d66a39d65"
Oct 30 17:45:18.828732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.828282    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"]
Oct 30 17:45:18.837011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.836985    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889705-99v8c"]
Oct 30 17:45:18.954673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.950704    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"]
Oct 30 17:45:18.956893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:18.956850    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889705-44h5j"]
Oct 30 17:45:19.036250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.036222    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:19.126067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.124848    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca\") pod \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") "
Oct 30 17:45:19.126067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.124977    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql\" (UniqueName: \"kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql\") pod \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\" (UID: \"2e1f04ed-4c86-4c75-bdc2-20aa12564d40\") "
Oct 30 17:45:19.126908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.126879    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca" pod "2e1f04ed-4c86-4c75-bdc2-20aa12564d40" (UID: "2e1f04ed-4c86-4c75-bdc2-20aa12564d40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:19.128278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.128242    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql" pod "2e1f04ed-4c86-4c75-bdc2-20aa12564d40" (UID: "2e1f04ed-4c86-4c75-bdc2-20aa12564d40"). InnerVolumeSpecName "kube-api-access-qw5ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:19.261565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.261252    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:19.261565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.261284    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw5ql\" (UniqueName: \"kubernetes.io/projected/2e1f04ed-4c86-4c75-bdc2-20aa12564d40-kube-api-access-qw5ql\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:19.372668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.366842    5534 scope.go:122] "RemoveContainer" containerID="22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8"
Oct 30 17:45:19.730702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.730512    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:19.794336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.793754    5534 generic.go:356] "Generic (PLEG): container finished" podID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerID="adf5cd9b265b8af5bc20eba7dd5b039a0dc9507735dd9992e2ec49fe793526a2" exitCode=60
Oct 30 17:45:19.794336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.793900    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerDied","Data":"adf5cd9b265b8af5bc20eba7dd5b039a0dc9507735dd9992e2ec49fe793526a2"}
Oct 30 17:45:19.794336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.793962    5534 scope.go:122] "RemoveContainer" containerID="c2cf2c3c44a8fa8b7c954bc21bfdca98970ec757e717b3d7aa112ede3668ad4e"
Oct 30 17:45:19.794684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.794511    5534 scope.go:122] "RemoveContainer" containerID="adf5cd9b265b8af5bc20eba7dd5b039a0dc9507735dd9992e2ec49fe793526a2"
Oct 30 17:45:19.796670 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:19.794977    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4_odh-ai-gateway-infra(aac47d37-9668-42b9-a8c7-e24eb5d1afa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9"
Oct 30 17:45:19.806389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.805015    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerStarted","Data":"9af6c1327c3a1eeed5bc1e19050e16c5b9bf16b22adb5c88ba85c894a827b837"}
Oct 30 17:45:19.809100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.809069    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerStarted","Data":"4028768026c933efd0c2edb6ab4ae67c17998c7e1fad42359c98b617be186833"}
Oct 30 17:45:19.828293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.827165    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerStarted","Data":"622ba8e45ecccd1d5854e7eb48d214c3fcccf695b37822995dd521252d5ae8f4"}
Oct 30 17:45:19.840535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.829148    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerStarted","Data":"6704877a514f9ade690e608c1be77efbc9ddd2457f0360d06c36e6f44ccd4d6b"}
Oct 30 17:45:19.960901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.960199    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt" event={"ID":"2e1f04ed-4c86-4c75-bdc2-20aa12564d40","Type":"ContainerDied","Data":"8e7be96064dbfbaa8c8c5fe185d4a6c1a5bda58cbed38d8c08b77f252fa0cc8b"}
Oct 30 17:45:19.960901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:19.960318    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"
Oct 30 17:45:20.043285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.043250    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"]
Oct 30 17:45:20.062217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.062176    5534 scope.go:122] "RemoveContainer" containerID="182602903d2041b207e450a28bb7a403bab50e765ef2bed50d76df847761d45c"
Oct 30 17:45:20.218335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.218304    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"]
Oct 30 17:45:20.237550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.237501    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889705-7kbpt"]
Oct 30 17:45:20.314494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.314460    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:20.427667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.427409    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5\" (UniqueName: \"kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5\") pod \"83712796-730a-494d-9fd3-805217d6823e\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") "
Oct 30 17:45:20.427667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.427586    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca\") pod \"83712796-730a-494d-9fd3-805217d6823e\" (UID: \"83712796-730a-494d-9fd3-805217d6823e\") "
Oct 30 17:45:20.442672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.435869    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca" pod "83712796-730a-494d-9fd3-805217d6823e" (UID: "83712796-730a-494d-9fd3-805217d6823e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:20.468711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.454898    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:20.468711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.461193    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5" pod "83712796-730a-494d-9fd3-805217d6823e" (UID: "83712796-730a-494d-9fd3-805217d6823e"). InnerVolumeSpecName "kube-api-access-qb7j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:20.538678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.533324    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83712796-730a-494d-9fd3-805217d6823e-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:20.538678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.533366    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7j5\" (UniqueName: \"kubernetes.io/projected/83712796-730a-494d-9fd3-805217d6823e-kube-api-access-qb7j5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:20.538678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.536190    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" path="/var/lib/kubelet/pods/2e1f04ed-4c86-4c75-bdc2-20aa12564d40/volumes"
Oct 30 17:45:20.604021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.603965    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" path="/var/lib/kubelet/pods/3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d/volumes"
Oct 30 17:45:20.605257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.605229    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" path="/var/lib/kubelet/pods/85827e4b-8f28-4b11-a1bb-b8b78dd94d43/volumes"
Oct 30 17:45:20.613395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.606246    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" path="/var/lib/kubelet/pods/f2a63f21-1afe-421e-9b25-91c082d51442/volumes"
Oct 30 17:45:20.805359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.805314    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:20.874848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:20.874791    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"]
Oct 30 17:45:21.080732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.079701    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"]
Oct 30 17:45:21.084399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.084360    5534 generic.go:356] "Generic (PLEG): container finished" podID="95e9bdd1-ed4d-493a-b112-e24721540858" containerID="6704877a514f9ade690e608c1be77efbc9ddd2457f0360d06c36e6f44ccd4d6b" exitCode=60
Oct 30 17:45:21.084909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.084868    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerDied","Data":"6704877a514f9ade690e608c1be77efbc9ddd2457f0360d06c36e6f44ccd4d6b"}
Oct 30 17:45:21.104085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.104059    5534 scope.go:122] "RemoveContainer" containerID="6c053ed9d2221afeb16a25f4c8bfc9c72a649019fb52b7126fc98fb23ccab906"
Oct 30 17:45:21.184925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.184889    5534 generic.go:356] "Generic (PLEG): container finished" podID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerID="59ed792b175a5370ec7d81c7078f39f4460c28c7e6d6410fcd9806bfbfd17afa" exitCode=60
Oct 30 17:45:21.185738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.185429    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerDied","Data":"59ed792b175a5370ec7d81c7078f39f4460c28c7e6d6410fcd9806bfbfd17afa"}
Oct 30 17:45:21.188506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.188478    5534 generic.go:356] "Generic (PLEG): container finished" podID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerID="9af6c1327c3a1eeed5bc1e19050e16c5b9bf16b22adb5c88ba85c894a827b837" exitCode=60
Oct 30 17:45:21.188741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.188718    5534 scope.go:122] "RemoveContainer" containerID="59ed792b175a5370ec7d81c7078f39f4460c28c7e6d6410fcd9806bfbfd17afa"
Oct 30 17:45:21.188873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.188855    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerDied","Data":"9af6c1327c3a1eeed5bc1e19050e16c5b9bf16b22adb5c88ba85c894a827b837"}
Oct 30 17:45:21.191889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.191860    5534 generic.go:356] "Generic (PLEG): container finished" podID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerID="4028768026c933efd0c2edb6ab4ae67c17998c7e1fad42359c98b617be186833" exitCode=60
Oct 30 17:45:21.192185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.192161    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerDied","Data":"4028768026c933efd0c2edb6ab4ae67c17998c7e1fad42359c98b617be186833"}
Oct 30 17:45:21.195693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.195633    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6" event={"ID":"83712796-730a-494d-9fd3-805217d6823e","Type":"ContainerDied","Data":"ee6b273a27d59c57cadba26839e99c5293d743b312ae1dc031bedcc184274dda"}
Oct 30 17:45:21.195985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.195965    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"
Oct 30 17:45:21.202167 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:21.202122    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs_odh-ai-gateway-infra(9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a"
Oct 30 17:45:21.202413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.202391    5534 scope.go:122] "RemoveContainer" containerID="9af6c1327c3a1eeed5bc1e19050e16c5b9bf16b22adb5c88ba85c894a827b837"
Oct 30 17:45:21.202764 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:21.202738    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49_odh-ai-gateway-infra(05d7ba6a-9ead-4270-a755-2ccbec88707e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e"
Oct 30 17:45:21.202885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.202784    5534 scope.go:122] "RemoveContainer" containerID="4028768026c933efd0c2edb6ab4ae67c17998c7e1fad42359c98b617be186833"
Oct 30 17:45:21.203050 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:21.203022    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8_odh-ai-gateway-infra(6d3b6fe8-94ca-444d-90ef-78aa4bbccf58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58"
Oct 30 17:45:21.207259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.207231    5534 generic.go:356] "Generic (PLEG): container finished" podID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerID="622ba8e45ecccd1d5854e7eb48d214c3fcccf695b37822995dd521252d5ae8f4" exitCode=60
Oct 30 17:45:21.207433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.207358    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerDied","Data":"622ba8e45ecccd1d5854e7eb48d214c3fcccf695b37822995dd521252d5ae8f4"}
Oct 30 17:45:21.208001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.207985    5534 scope.go:122] "RemoveContainer" containerID="622ba8e45ecccd1d5854e7eb48d214c3fcccf695b37822995dd521252d5ae8f4"
Oct 30 17:45:21.208692 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:21.208668    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq_odh-ai-gateway-infra(d7a5d82b-7418-42a6-8265-1a889f3ebecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf"
Oct 30 17:45:21.238875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.238831    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"]
Oct 30 17:45:21.259592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.258984    5534 scope.go:122] "RemoveContainer" containerID="22ff086235c689fde98e77361fe2166490f0678e950ba50107ec98ccf10080b8"
Oct 30 17:45:21.361842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.355742    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"]
Oct 30 17:45:21.550663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.550595    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"]
Oct 30 17:45:21.801900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:21.797352    5534 scope.go:122] "RemoveContainer" containerID="d03043abeff9f73f7c347d15ee25f00fdc8ec50281069a3139c7e20db47280c1"
Oct 30 17:45:22.012050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.012016    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"]
Oct 30 17:45:22.050351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.050287    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889705-fz6h6"]
Oct 30 17:45:22.059097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.058980    5534 scope.go:122] "RemoveContainer" containerID="ddae182a36e5984ec97bf4b0b6bacc1d099f54e8438724107a71630a91a75378"
Oct 30 17:45:22.096285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.095888    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:22.119765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.119717    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2\" (UniqueName: \"kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2\") pod \"95e9bdd1-ed4d-493a-b112-e24721540858\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") "
Oct 30 17:45:22.120274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.120249    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca\") pod \"95e9bdd1-ed4d-493a-b112-e24721540858\" (UID: \"95e9bdd1-ed4d-493a-b112-e24721540858\") "
Oct 30 17:45:22.121010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.120964    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca" pod "95e9bdd1-ed4d-493a-b112-e24721540858" (UID: "95e9bdd1-ed4d-493a-b112-e24721540858"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:22.122076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.122056    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9bdd1-ed4d-493a-b112-e24721540858-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:22.133042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.133001    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2" pod "95e9bdd1-ed4d-493a-b112-e24721540858" (UID: "95e9bdd1-ed4d-493a-b112-e24721540858"). InnerVolumeSpecName "kube-api-access-fpzh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:22.139179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.139037    5534 scope.go:122] "RemoveContainer" containerID="6feee743922c4dda803d7ed4dd0e2a09422ca7e442f3a6181d47cce3df5d6704"
Oct 30 17:45:22.208144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.208110    5534 scope.go:122] "RemoveContainer" containerID="74ca2cd3dd35a8129e04fc77c30c4d3a3ccef15aff41e622369a5ff34cfa1cee"
Oct 30 17:45:22.226969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.226933    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpzh2\" (UniqueName: \"kubernetes.io/projected/95e9bdd1-ed4d-493a-b112-e24721540858-kube-api-access-fpzh2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:22.250300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.249903    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:22.332157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.332084    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2\" (UniqueName: \"kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2\") pod \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") "
Oct 30 17:45:22.345974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.345188    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca\") pod \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\" (UID: \"aac47d37-9668-42b9-a8c7-e24eb5d1afa9\") "
Oct 30 17:45:22.347139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.346358    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2" pod "aac47d37-9668-42b9-a8c7-e24eb5d1afa9" (UID: "aac47d37-9668-42b9-a8c7-e24eb5d1afa9"). InnerVolumeSpecName "kube-api-access-49jt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:22.347331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.347299    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca" pod "aac47d37-9668-42b9-a8c7-e24eb5d1afa9" (UID: "aac47d37-9668-42b9-a8c7-e24eb5d1afa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:22.347951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.347908    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jt2\" (UniqueName: \"kubernetes.io/projected/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-kube-api-access-49jt2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:22.348055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.347957    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac47d37-9668-42b9-a8c7-e24eb5d1afa9-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:22.380781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.350299    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"
Oct 30 17:45:22.380781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.361762    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k" event={"ID":"95e9bdd1-ed4d-493a-b112-e24721540858","Type":"ContainerDied","Data":"bbfa22129e310c07e7e03b19c8ef147f053fe08d6ec0cc7cf896f43542a38712"}
Oct 30 17:45:22.380781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.361850    5534 scope.go:122] "RemoveContainer" containerID="6704877a514f9ade690e608c1be77efbc9ddd2457f0360d06c36e6f44ccd4d6b"
Oct 30 17:45:22.447018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.446977    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83712796-730a-494d-9fd3-805217d6823e" path="/var/lib/kubelet/pods/83712796-730a-494d-9fd3-805217d6823e/volumes"
Oct 30 17:45:22.580427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.580334    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"]
Oct 30 17:45:22.635692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.632027    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"]
Oct 30 17:45:22.639302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.639270    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889705-8259k"]
Oct 30 17:45:22.643730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.643365    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:22.918620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:22.918588    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:23.097832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.064828    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca\") pod \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") "
Oct 30 17:45:23.097832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.064893    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh\" (UniqueName: \"kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh\") pod \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\" (UID: \"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58\") "
Oct 30 17:45:23.098890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.098850    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca" pod "6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" (UID: "6d3b6fe8-94ca-444d-90ef-78aa4bbccf58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:23.116819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.116781    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh" pod "6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" (UID: "6d3b6fe8-94ca-444d-90ef-78aa4bbccf58"). InnerVolumeSpecName "kube-api-access-6bqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:23.209490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.209388    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.209722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.209702    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqmh\" (UniqueName: \"kubernetes.io/projected/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58-kube-api-access-6bqmh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.219661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.219529    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:23.228031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.227632    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:23.311816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.311057    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:23.311816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.311088    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw\" (UniqueName: \"kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw\") pod \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") "
Oct 30 17:45:23.311816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.311253    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca\") pod \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\" (UID: \"d7a5d82b-7418-42a6-8265-1a889f3ebecf\") "
Oct 30 17:45:23.311816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.311398    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca\") pod \"05d7ba6a-9ead-4270-a755-2ccbec88707e\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") "
Oct 30 17:45:23.311816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.311518    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r\" (UniqueName: \"kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r\") pod \"05d7ba6a-9ead-4270-a755-2ccbec88707e\" (UID: \"05d7ba6a-9ead-4270-a755-2ccbec88707e\") "
Oct 30 17:45:23.311816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.311818    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca" pod "d7a5d82b-7418-42a6-8265-1a889f3ebecf" (UID: "d7a5d82b-7418-42a6-8265-1a889f3ebecf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:23.312237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.312192    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a5d82b-7418-42a6-8265-1a889f3ebecf-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.312473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.312449    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca" pod "05d7ba6a-9ead-4270-a755-2ccbec88707e" (UID: "05d7ba6a-9ead-4270-a755-2ccbec88707e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:23.316730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.314061    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw" pod "d7a5d82b-7418-42a6-8265-1a889f3ebecf" (UID: "d7a5d82b-7418-42a6-8265-1a889f3ebecf"). InnerVolumeSpecName "kube-api-access-k65vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:23.316730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.315115    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r" pod "05d7ba6a-9ead-4270-a755-2ccbec88707e" (UID: "05d7ba6a-9ead-4270-a755-2ccbec88707e"). InnerVolumeSpecName "kube-api-access-mld2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:23.413713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.413687    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca\") pod \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") "
Oct 30 17:45:23.413941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.413928    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt\" (UniqueName: \"kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt\") pod \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\" (UID: \"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a\") "
Oct 30 17:45:23.416768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.414676    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca" pod "9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" (UID: "9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:23.417200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.417168    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49" event={"ID":"05d7ba6a-9ead-4270-a755-2ccbec88707e","Type":"ContainerDied","Data":"0e128c969b38bf562ba1dd5125f20ac61f2962c36e125e314eb2373c2740c012"}
Oct 30 17:45:23.419875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.419747    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8" event={"ID":"6d3b6fe8-94ca-444d-90ef-78aa4bbccf58","Type":"ContainerDied","Data":"8c18eddfb319285ee3e5f3c47b7a2080075d3dc412f815874dd0efe16b920dd2"}
Oct 30 17:45:23.419875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.419318    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"
Oct 30 17:45:23.420191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.417949    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"
Oct 30 17:45:23.420277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.420263    5534 scope.go:122] "RemoveContainer" containerID="9af6c1327c3a1eeed5bc1e19050e16c5b9bf16b22adb5c88ba85c894a827b837"
Oct 30 17:45:23.420350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.419501    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt" pod "9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" (UID: "9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a"). InnerVolumeSpecName "kube-api-access-6nhwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:23.421129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.421104    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05d7ba6a-9ead-4270-a755-2ccbec88707e-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.421217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.421134    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mld2r\" (UniqueName: \"kubernetes.io/projected/05d7ba6a-9ead-4270-a755-2ccbec88707e-kube-api-access-mld2r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.421217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.421150    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.421217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.421163    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65vw\" (UniqueName: \"kubernetes.io/projected/d7a5d82b-7418-42a6-8265-1a889f3ebecf-kube-api-access-k65vw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.421217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.421178    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhwt\" (UniqueName: \"kubernetes.io/projected/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a-kube-api-access-6nhwt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:23.433749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.433682    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq" event={"ID":"d7a5d82b-7418-42a6-8265-1a889f3ebecf","Type":"ContainerDied","Data":"5a94fb39b01d7500ce141b8fddeeb925fd7f1ecfee0a9861a9630ee4f75d1781"}
Oct 30 17:45:23.434144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.434073    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"
Oct 30 17:45:23.474181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.474145    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4" event={"ID":"aac47d37-9668-42b9-a8c7-e24eb5d1afa9","Type":"ContainerDied","Data":"ab2e9dea85a14258fcc8362fa1498a5acd728b68981bd657dda3b80281a3ef70"}
Oct 30 17:45:23.474550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.474532    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"
Oct 30 17:45:23.497005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.496959    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs" event={"ID":"9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a","Type":"ContainerDied","Data":"92d77b2d6354e5336284b0342b5cee65fa49287634002405f752a56dcbe54138"}
Oct 30 17:45:23.497210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.497098    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"
Oct 30 17:45:23.511854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.511720    5534 scope.go:122] "RemoveContainer" containerID="4028768026c933efd0c2edb6ab4ae67c17998c7e1fad42359c98b617be186833"
Oct 30 17:45:23.529411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.529377    5534 scope.go:122] "RemoveContainer" containerID="622ba8e45ecccd1d5854e7eb48d214c3fcccf695b37822995dd521252d5ae8f4"
Oct 30 17:45:23.533082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.533055    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"]
Oct 30 17:45:23.538112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.538086    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889705-x8wx8"]
Oct 30 17:45:23.546445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.546423    5534 scope.go:122] "RemoveContainer" containerID="adf5cd9b265b8af5bc20eba7dd5b039a0dc9507735dd9992e2ec49fe793526a2"
Oct 30 17:45:23.604431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.604402    5534 scope.go:122] "RemoveContainer" containerID="59ed792b175a5370ec7d81c7078f39f4460c28c7e6d6410fcd9806bfbfd17afa"
Oct 30 17:45:23.612286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.612259    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"]
Oct 30 17:45:23.618929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.618900    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889705-z9t49"]
Oct 30 17:45:23.647923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.647891    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"]
Oct 30 17:45:23.651872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.651842    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889705-577mq"]
Oct 30 17:45:23.701288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.701245    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"]
Oct 30 17:45:23.707600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.707548    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ee9ff3-29889705-5s4k4"]
Oct 30 17:45:23.743913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.743828    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"]
Oct 30 17:45:23.748793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:23.748756    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889705-h88fs"]
Oct 30 17:45:24.407049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.407008    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" path="/var/lib/kubelet/pods/05d7ba6a-9ead-4270-a755-2ccbec88707e/volumes"
Oct 30 17:45:24.408449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.408423    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" path="/var/lib/kubelet/pods/6d3b6fe8-94ca-444d-90ef-78aa4bbccf58/volumes"
Oct 30 17:45:24.409382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.409363    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" path="/var/lib/kubelet/pods/95e9bdd1-ed4d-493a-b112-e24721540858/volumes"
Oct 30 17:45:24.410682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.410633    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" path="/var/lib/kubelet/pods/9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a/volumes"
Oct 30 17:45:24.411670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.411635    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" path="/var/lib/kubelet/pods/aac47d37-9668-42b9-a8c7-e24eb5d1afa9/volumes"
Oct 30 17:45:24.412470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.412452    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" path="/var/lib/kubelet/pods/d7a5d82b-7418-42a6-8265-1a889f3ebecf/volumes"
Oct 30 17:45:24.523528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.523461    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:24.636144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.636103    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:24.788858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.778822    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:24.849471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.849281    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"]
Oct 30 17:45:24.849801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.849753    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" containerID="cri-o://f339d098e6641e3f7fde6fa66bf1c76fcca481dfb19228b0c85ac93196a8b196" gracePeriod=30
Oct 30 17:45:24.903301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:24.901561    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:26.036037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:26.036001    5534 patch_prober.go:29] interesting pod/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:45:26.036559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:26.036064    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:45:26.699547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:26.699509    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:27.739823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:27.739777    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:45:27.740290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:27.739843    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:45:28.600732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:28.600632    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:29.834077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:29.834037    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:30.395477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.395443    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:30.588608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.588173    5534 generic.go:356] "Generic (PLEG): container finished" podID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerID="f339d098e6641e3f7fde6fa66bf1c76fcca481dfb19228b0c85ac93196a8b196" exitCode=0
Oct 30 17:45:30.588841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.588692    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" event={"ID":"af1f1ddd-0676-4413-9dd2-1ad060ec5090","Type":"ContainerDied","Data":"f339d098e6641e3f7fde6fa66bf1c76fcca481dfb19228b0c85ac93196a8b196"}
Oct 30 17:45:30.588841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.588778    5534 scope.go:122] "RemoveContainer" containerID="ff695258dd354590231a26c6c6e6a8130ed7971cd2d60abb5639d2e785887f4d"
Oct 30 17:45:30.700223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.700185    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:45:30.847683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847605    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847736    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847868    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847901    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847928    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847923    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847957    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.847984    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.848127    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.848196    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token\") pod \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\" (UID: \"af1f1ddd-0676-4413-9dd2-1ad060ec5090\") "
Oct 30 17:45:30.848763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.848236    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:45:30.848763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.848252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:30.848763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.848519    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:45:30.849285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.849086    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.849285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.849147    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:45:30.849285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.849154    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.849285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.849193    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.849285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.849205    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.850921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.850892    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:45:30.851063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.850921    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:30.851063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.850981    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:45:30.851205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.851183    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4" pod "af1f1ddd-0676-4413-9dd2-1ad060ec5090" (UID: "af1f1ddd-0676-4413-9dd2-1ad060ec5090"). InnerVolumeSpecName "kube-api-access-62jl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:30.951407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.951313    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.951407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.951348    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.951407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.951359    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62jl4\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-kube-api-access-62jl4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.951407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.951372    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1f1ddd-0676-4413-9dd2-1ad060ec5090-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:30.951407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:30.951387    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1f1ddd-0676-4413-9dd2-1ad060ec5090-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:31.598939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.598894    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerID="fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a" exitCode=0
Oct 30 17:45:31.599150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.599046    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" event={"ID":"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10","Type":"ContainerDied","Data":"fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"}
Oct 30 17:45:31.599602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.599580    5534 scope.go:122] "RemoveContainer" containerID="fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"
Oct 30 17:45:31.604879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.604842    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:31.605033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.604939    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:31.605033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.604994    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:31.607952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.605885    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"
Oct 30 17:45:31.607952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.605921    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4" event={"ID":"af1f1ddd-0676-4413-9dd2-1ad060ec5090","Type":"ContainerDied","Data":"ecb060e51180abb6db736587d7675cde11181aa23e6a8ec365840f51aa05f0f1"}
Oct 30 17:45:31.607952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.605997    5534 scope.go:122] "RemoveContainer" containerID="f339d098e6641e3f7fde6fa66bf1c76fcca481dfb19228b0c85ac93196a8b196"
Oct 30 17:45:31.704595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.704551    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"]
Oct 30 17:45:31.710706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:31.710681    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cc4e96-openshift-default-c6c887c8c-rndb4"]
Oct 30 17:45:32.193537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:32.193500    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:32.400703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:32.400628    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" path="/var/lib/kubelet/pods/af1f1ddd-0676-4413-9dd2-1ad060ec5090/volumes"
Oct 30 17:45:32.619774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:32.619735    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" event={"ID":"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10","Type":"ContainerStarted","Data":"c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1"}
Oct 30 17:45:33.244742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:33.244695    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:45:33.245284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:33.244983    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 17:45:33.245284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:33.245066    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 17:45:33.904319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:33.904280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:34.245872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:34.245781    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 17:45:34.245872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:34.245855    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 17:45:34.424392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:34.424361    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:35.245478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:35.245439    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 17:45:35.245689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:35.245508    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 17:45:35.788667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:35.786571    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:35.919230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:35.918750    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:36.245738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.245706    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 17:45:36.245934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.245769    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 17:45:36.303789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.279751    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"]
Oct 30 17:45:36.306048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306020    5534 cpu_manager.go:414] "containerMap: removing container" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:36.306281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306267    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:36.306385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306376    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.306458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306449    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.306545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306537    5534 cpu_manager.go:414] "containerMap: removing container" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:36.306625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306615    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:36.306733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306722    5534 cpu_manager.go:414] "containerMap: removing container" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy"
Oct 30 17:45:36.306817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.306807    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307686    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307707    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307730    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307738    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307747    5534 cpu_manager.go:414] "containerMap: removing container" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307756    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307776    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307783    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307796    5534 cpu_manager.go:414] "containerMap: removing container" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307803    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307826    5534 cpu_manager.go:414] "containerMap: removing container" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307832    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307846    5534 cpu_manager.go:414] "containerMap: removing container" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307853    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307866    5534 cpu_manager.go:414] "containerMap: removing container" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307873    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307881    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307888    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307901    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307918    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307935    5534 cpu_manager.go:414] "containerMap: removing container" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307942    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307951    5534 cpu_manager.go:414] "containerMap: removing container" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307959    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307967    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307974    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307988    5534 cpu_manager.go:414] "containerMap: removing container" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.307995    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308010    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308017    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308027    5534 cpu_manager.go:414] "containerMap: removing container" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308035    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308060    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308067    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308077    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:36.308074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308084    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308105    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308113    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308128    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308135    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308151    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308157    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308172    5534 cpu_manager.go:414] "containerMap: removing container" podUID="78ba9c34-8c81-4b68-90dc-baa751b5cb93" containerName="payload-processing"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308178    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ba9c34-8c81-4b68-90dc-baa751b5cb93" containerName="payload-processing"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308197    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308204    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308222    5534 cpu_manager.go:414] "containerMap: removing container" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308229    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308476    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308490    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308499    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308510    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308518    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308528    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308536    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308545    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308554    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308563    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308572    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308579    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308587    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308597    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308604    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="83712796-730a-494d-9fd3-805217d6823e" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308615    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308625    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308634    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308674    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308684    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308693    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308702    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308713    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ba9c34-8c81-4b68-90dc-baa751b5cb93" containerName="payload-processing"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308721    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308730    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308740    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308749    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308762    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:36.310307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.308771    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy"
Oct 30 17:45:36.313742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.313718    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"]
Oct 30 17:45:36.313877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.313843    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.329069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.326345    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:36.334219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.334169    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-043253-openshift-default-dockercfg-vlpqj\""
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.386295    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.386431    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqwz\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.386538    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.386706    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.386862    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.386922    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.387112    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.387177    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.392362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.387206    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.504675    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.504795    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.504836    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.504939    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.504972    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.504999    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.505119    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.505185    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqwz\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.505304    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.505825    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.506728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.506266    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.528686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.523887    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.543444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.542114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:36.543444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.543219    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.544243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.544191    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.600020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.596033    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.600020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.599960    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqwz\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.604277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.601322    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.604277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.604210    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bczqh\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:36.642197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:36.642087    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:37.099095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.091680    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"]
Oct 30 17:45:37.102716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.102678    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:37.102867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.102759    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:37.102867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.102793    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:45:37.245338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.245295    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 17:45:37.245508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.245364    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 17:45:37.741714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.741673    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" event={"ID":"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f","Type":"ContainerStarted","Data":"4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"}
Oct 30 17:45:37.741927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.741745    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" event={"ID":"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f","Type":"ContainerStarted","Data":"c1c0ae4ff3760f893712f2dffb2f1cf0e360c8d07488dc5ac81770563bcdd560"}
Oct 30 17:45:37.809509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.809443    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podStartSLOduration=1.809426422 podStartE2EDuration="1.809426422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:37.808971438 +0000 UTC m=+1943.571015272" watchObservedRunningTime="2026-10-30 17:45:37.809426422 +0000 UTC m=+1943.571470235"
Oct 30 17:45:37.969678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:37.969612    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:38.249854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.249822    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:45:38.250329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.250315    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:45:38.250985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.250966    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:45:38.643577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.643543    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:38.651172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.651141    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:38.756627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.756592    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:38.757726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:38.757703    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:45:39.665855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:39.665821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:41.324622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:41.324588    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:42.982220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:42.982187    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:44.678299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:44.678260    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:45.247755 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:45.247717    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:45:46.310760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:46.310726    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:48.197688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:48.197627    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:50.139909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:50.139878    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:50.528728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:50.528592    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:50.726073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:50.726026    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:50.852492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:50.852406    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:52.578226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.578197    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:52.709069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709025    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"]
Oct 30 17:45:52.709810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709793    5534 cpu_manager.go:414] "containerMap: removing container" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy"
Oct 30 17:45:52.709810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709809    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1f1ddd-0676-4413-9dd2-1ad060ec5090" containerName="istio-proxy"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709825    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709830    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f15f5e4-aa84-478e-9111-bff23ac5463a" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709839    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709844    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709851    5534 cpu_manager.go:414] "containerMap: removing container" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709856    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709862    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709867    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709882    5534 cpu_manager.go:414] "containerMap: removing container" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709887    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709901    5534 cpu_manager.go:414] "containerMap: removing container" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709905    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709914    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709918    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709932    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709936    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709946    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:52.709986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.709950    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a5d82b-7418-42a6-8265-1a889f3ebecf" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710073    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d7ba6a-9ead-4270-a755-2ccbec88707e" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710085    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe835fa-e4df-4ca5-8f7c-b6aa498e3c6a" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710092    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3b6fe8-94ca-444d-90ef-78aa4bbccf58" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710099    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6e8a5a-e240-4f63-8e30-f2d5b5295b8d" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710114    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a63f21-1afe-421e-9b25-91c082d51442" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710120    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1f04ed-4c86-4c75-bdc2-20aa12564d40" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710128    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="85827e4b-8f28-4b11-a1bb-b8b78dd94d43" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710148    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e9bdd1-ed4d-493a-b112-e24721540858" containerName="cleanup"
Oct 30 17:45:52.710934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.710159    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac47d37-9668-42b9-a8c7-e24eb5d1afa9" containerName="cleanup"
Oct 30 17:45:52.714358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.714335    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:52.735768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.735740    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:52.738454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.738426    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"]
Oct 30 17:45:52.745119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.745090    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-043253\""
Oct 30 17:45:52.745571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.745532    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-043253-dockercfg-fpxr8\""
Oct 30 17:45:52.828716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.825625    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rb6q\" (UniqueName: \"kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q\") pod \"payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:52.828716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.825733    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:52.945831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.933615    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rb6q\" (UniqueName: \"kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q\") pod \"payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:52.962815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.962771    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:52.975667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:52.964485    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:53.029632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.010829    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rb6q\" (UniqueName: \"kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q\") pod \"payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:53.031169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.031144    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:53.065712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.041977    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"]
Oct 30 17:45:53.101666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.098733    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.101666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.099405    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"]
Oct 30 17:45:53.174751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.171552    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27v6\" (UniqueName: \"kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6\") pod \"payload-processing-e2e-inf-a-043253-8587649dc4-49lwm\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") " pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.174751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.171822    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-043253-8587649dc4-49lwm\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") " pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.273681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.273604    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27v6\" (UniqueName: \"kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6\") pod \"payload-processing-e2e-inf-a-043253-8587649dc4-49lwm\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") " pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.273898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.273839    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-043253-8587649dc4-49lwm\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") " pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.274935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.274910    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-043253-8587649dc4-49lwm\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") " pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.297417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.297377    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27v6\" (UniqueName: \"kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6\") pod \"payload-processing-e2e-inf-a-043253-8587649dc4-49lwm\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") " pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.391215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.391173    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"]
Oct 30 17:45:53.397841 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:53.397810    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac24c3e_c7d1_495a_8b4d_8a87d28087a8.slice/crio-3b65ec07f0803175cfae4519d308c1550b63a8a7d910510301485a46eb373c1f WatchSource:0}: Error finding container 3b65ec07f0803175cfae4519d308c1550b63a8a7d910510301485a46eb373c1f: Status 404 returned error can't find the container with id 3b65ec07f0803175cfae4519d308c1550b63a8a7d910510301485a46eb373c1f
Oct 30 17:45:53.420533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.420490    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:53.753242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:53.753208    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"]
Oct 30 17:45:53.753705 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:45:53.753492    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b07c046_bdbd_451c_83d6_1b8e50b1b010.slice/crio-3252dbba3718cdcb9002600bf02e64126152fc0e3fa38e4e170b21d6634db8d5 WatchSource:0}: Error finding container 3252dbba3718cdcb9002600bf02e64126152fc0e3fa38e4e170b21d6634db8d5: Status 404 returned error can't find the container with id 3252dbba3718cdcb9002600bf02e64126152fc0e3fa38e4e170b21d6634db8d5
Oct 30 17:45:54.062870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.062789    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"]
Oct 30 17:45:54.063053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.063028    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw" podUID="6d0c353c-77b2-4b87-a22c-9e723a6de0ca" containerName="maas-api" containerID="cri-o://73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834" gracePeriod=30
Oct 30 17:45:54.112710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.112673    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" event={"ID":"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8","Type":"ContainerStarted","Data":"9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5"}
Oct 30 17:45:54.112968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.112947    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:45:54.113107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.113090    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" event={"ID":"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8","Type":"ContainerStarted","Data":"3b65ec07f0803175cfae4519d308c1550b63a8a7d910510301485a46eb373c1f"}
Oct 30 17:45:54.114624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.114600    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm" event={"ID":"6b07c046-bdbd-451c-83d6-1b8e50b1b010","Type":"ContainerStarted","Data":"289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c"}
Oct 30 17:45:54.114772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.114660    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm" event={"ID":"6b07c046-bdbd-451c-83d6-1b8e50b1b010","Type":"ContainerStarted","Data":"3252dbba3718cdcb9002600bf02e64126152fc0e3fa38e4e170b21d6634db8d5"}
Oct 30 17:45:54.114772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.114688    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:45:54.135002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.134972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:54.155960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.155911    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" podStartSLOduration=2.155897493 podStartE2EDuration="2.155897493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:54.152950846 +0000 UTC m=+1959.914994662" watchObservedRunningTime="2026-10-30 17:45:54.155897493 +0000 UTC m=+1959.917941319"
Oct 30 17:45:54.196353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.196303    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm" podStartSLOduration=2.19628819 podStartE2EDuration="2.19628819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:54.193216124 +0000 UTC m=+1959.955259937" watchObservedRunningTime="2026-10-30 17:45:54.19628819 +0000 UTC m=+1959.958332024"
Oct 30 17:45:54.562200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.561842    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:45:54.743273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.743161    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca\") pod \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") "
Oct 30 17:45:54.743473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.743370    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls\") pod \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") "
Oct 30 17:45:54.743473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.743417    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs\") pod \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") "
Oct 30 17:45:54.743595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.743519    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb\" (UniqueName: \"kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb\") pod \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\" (UID: \"6d0c353c-77b2-4b87-a22c-9e723a6de0ca\") "
Oct 30 17:45:54.744167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.744138    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca" pod "6d0c353c-77b2-4b87-a22c-9e723a6de0ca" (UID: "6d0c353c-77b2-4b87-a22c-9e723a6de0ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:54.744293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.744268    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:54.748017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.747988    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls" pod "6d0c353c-77b2-4b87-a22c-9e723a6de0ca" (UID: "6d0c353c-77b2-4b87-a22c-9e723a6de0ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:45:54.749812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.749781    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs" pod "6d0c353c-77b2-4b87-a22c-9e723a6de0ca" (UID: "6d0c353c-77b2-4b87-a22c-9e723a6de0ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:45:54.749937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.749862    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb" pod "6d0c353c-77b2-4b87-a22c-9e723a6de0ca" (UID: "6d0c353c-77b2-4b87-a22c-9e723a6de0ca"). InnerVolumeSpecName "kube-api-access-fwxpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:54.757156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.757111    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:54.845398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.845363    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:54.845398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.845395    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:54.845398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.845404    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxpb\" (UniqueName: \"kubernetes.io/projected/6d0c353c-77b2-4b87-a22c-9e723a6de0ca-kube-api-access-fwxpb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:45:54.868071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.868035    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:54.888847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.888812    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"]
Oct 30 17:45:54.893472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.893443    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74762d-v2xdp"]
Oct 30 17:45:54.983900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:54.983850    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:55.135395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.135361    5534 generic.go:356] "Generic (PLEG): container finished" podID="6d0c353c-77b2-4b87-a22c-9e723a6de0ca" containerID="73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834" exitCode=0
Oct 30 17:45:55.135606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.135442    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw" event={"ID":"6d0c353c-77b2-4b87-a22c-9e723a6de0ca","Type":"ContainerDied","Data":"73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834"}
Oct 30 17:45:55.135606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.135447    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"
Oct 30 17:45:55.135606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.135539    5534 scope.go:122] "RemoveContainer" containerID="73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834"
Oct 30 17:45:55.135834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.135526    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw" event={"ID":"6d0c353c-77b2-4b87-a22c-9e723a6de0ca","Type":"ContainerDied","Data":"992ed019983794a9a4bc712ba1bd40c4c971c748f0d3dadf4cae01e062bde799"}
Oct 30 17:45:55.148100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.148077    5534 scope.go:122] "RemoveContainer" containerID="73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834"
Oct 30 17:45:55.148405 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:45:55.148383    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834\": container with ID starting with 73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834 not found: ID does not exist" containerID="73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834"
Oct 30 17:45:55.148464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.148415    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834"} err="failed to get container status \"73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834\": rpc error: code = NotFound desc = could not find container \"73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834\": container with ID starting with 73d5a262f7af2859a6e86da6717aa65f9c89d97040e0e45d3398bd41ab590834 not found: ID does not exist"
Oct 30 17:45:55.189922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.189889    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"]
Oct 30 17:45:55.201400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.201368    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74762d-5b8c677d75-4t8fw"]
Oct 30 17:45:55.455033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.454885    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:55.726151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:55.726059    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:56.375186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:56.375156    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453432ed-2dc8-461d-aca3-1a9c64ef8860" path="/var/lib/kubelet/pods/453432ed-2dc8-461d-aca3-1a9c64ef8860/volumes"
Oct 30 17:45:56.375749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:56.375620    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0c353c-77b2-4b87-a22c-9e723a6de0ca" path="/var/lib/kubelet/pods/6d0c353c-77b2-4b87-a22c-9e723a6de0ca/volumes"
Oct 30 17:45:56.915107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:56.915058    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:58.584617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:58.584568    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:45:58.736289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:45:58.736242    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:00.372327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:00.372293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:00.737261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:00.737141    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:02.001451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.001412    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:02.126001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.125962    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:02.243873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.243838    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:02.336973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.336824    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"]
Oct 30 17:46:02.339605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.339563    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" containerID="cri-o://560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05" gracePeriod=30
Oct 30 17:46:02.347546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.347511    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"]
Oct 30 17:46:02.347844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.347809    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy" containerID="cri-o://e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0" gracePeriod=30
Oct 30 17:46:02.386774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:02.384843    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:03.194248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.194181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:03.400183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.400149    5534 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:46:03.400354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.400226    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:46:03.617899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.617855    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"]
Oct 30 17:46:03.619465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.619436    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6d0c353c-77b2-4b87-a22c-9e723a6de0ca" containerName="maas-api"
Oct 30 17:46:03.619465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.619464    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0c353c-77b2-4b87-a22c-9e723a6de0ca" containerName="maas-api"
Oct 30 17:46:03.619726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.619700    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0c353c-77b2-4b87-a22c-9e723a6de0ca" containerName="maas-api"
Oct 30 17:46:03.624560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.624533    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.642470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.640815    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4a271c-openshift-default-dockercfg-46ql8\""
Oct 30 17:46:03.666395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.666355    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"]
Oct 30 17:46:03.688596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.688561    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:03.718343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718310    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.718343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718353    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.718624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718536    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5qf\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.718624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718609    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.718815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.718815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718778    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.718982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.718961    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.719045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.719028    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.719087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.719058    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.837179    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838334    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5qf\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838390    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838444    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838502    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838634    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838722    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838745    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838828    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.838856    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.843800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.839386    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.849677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.847335    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.849677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.847617    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.849677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.848266    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.849677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.848511    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.873716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.869308    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.891665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.880072    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.891665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.888930    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5qf\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.913672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.908034    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:03.950666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:03.945910    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:04.198398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:04.198294    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:04.340485 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:04.340448    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42639ec9_312c_49fa_b750_3cead79a0534.slice/crio-856a6ae4f7b3de9c36c79ff5338c907bb0ac8494c06c41601c1f20e692828fff WatchSource:0}: Error finding container 856a6ae4f7b3de9c36c79ff5338c907bb0ac8494c06c41601c1f20e692828fff: Status 404 returned error can't find the container with id 856a6ae4f7b3de9c36c79ff5338c907bb0ac8494c06c41601c1f20e692828fff
Oct 30 17:46:04.340603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:04.340561    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"]
Oct 30 17:46:04.343285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:04.343253    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:04.343349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:04.343320    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:04.343386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:04.343350    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:05.139659 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.139614    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:46:05.139884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.139710    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:46:05.258547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.258499    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" event={"ID":"42639ec9-312c-49fa-b750-3cead79a0534","Type":"ContainerStarted","Data":"737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"}
Oct 30 17:46:05.259473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.259438    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" event={"ID":"42639ec9-312c-49fa-b750-3cead79a0534","Type":"ContainerStarted","Data":"856a6ae4f7b3de9c36c79ff5338c907bb0ac8494c06c41601c1f20e692828fff"}
Oct 30 17:46:05.309173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.309109    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podStartSLOduration=2.309089862 podStartE2EDuration="2.309089862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:05.304149113 +0000 UTC m=+1971.066192926" watchObservedRunningTime="2026-10-30 17:46:05.309089862 +0000 UTC m=+1971.071133675"
Oct 30 17:46:05.947108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.947076    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:05.953429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:05.953387    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:06.081492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:06.081454    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:06.267973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:06.267875    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:06.268958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:06.268935    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:46:06.387216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:06.387177    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:07.657157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.657127    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:46:07.701741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.701704    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.701922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.701834    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.701922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.701894    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.701968    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702006    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702157 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702069    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702159    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.702213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702176    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702249    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702323 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702250    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:46:07.702429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702356    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy\") pod \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\" (UID: \"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b\") "
Oct 30 17:46:07.702429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702383    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.702574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702556    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.702749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.702730    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.703248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.703223    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.703248 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.703248    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.703457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.703262    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.703457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.703274    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.703457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.703288    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.704620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.704591    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:46:07.705034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.705012    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "kube-api-access-w2d8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:46:07.705133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.705116    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:46:07.705523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.705502    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy" pod "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" (UID: "88c0f38e-2d80-4cfb-9bc5-70dcaf27011b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.724606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.724569    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:46:07.795707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.785860    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:07.804691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.804636    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.804882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.804726    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.804882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.804816    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.804882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.804846    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.805069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.804916    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.805069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.805062    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.805191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.805110    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.805191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.805174    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.805302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.805276    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket\") pod \"4c9570be-a0d3-4b42-849d-9523b3f794f9\" (UID: \"4c9570be-a0d3-4b42-849d-9523b3f794f9\") "
Oct 30 17:46:07.805395 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.805370    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.805566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.805547    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.806151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806123    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.806267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806155    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2d8h\" (UniqueName: \"kubernetes.io/projected/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-kube-api-access-w2d8h\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.806267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806170    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.806267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806185    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.806267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806191    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:46:07.806267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806199    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.806267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806239    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.806614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806402    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.806614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.806436    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.807329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.807298    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:46:07.808657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.808614    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:46:07.808864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.808846    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "kube-api-access-dmxhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:46:07.809512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.809493    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy" pod "4c9570be-a0d3-4b42-849d-9523b3f794f9" (UID: "4c9570be-a0d3-4b42-849d-9523b3f794f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:46:07.907764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907659    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmxhm\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-kube-api-access-dmxhm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.907764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907700    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9570be-a0d3-4b42-849d-9523b3f794f9-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.907764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907718    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.907764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907732    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.907764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907746    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.907764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907764    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.908201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.907778    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9570be-a0d3-4b42-849d-9523b3f794f9-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:46:07.929658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:07.929606    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:08.098749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.095880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:08.220682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.220575    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:08.287064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.287031    5534 generic.go:356] "Generic (PLEG): container finished" podID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerID="560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05" exitCode=0
Oct 30 17:46:08.287287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.287099    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"
Oct 30 17:46:08.287287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.287094    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" event={"ID":"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b","Type":"ContainerDied","Data":"560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05"}
Oct 30 17:46:08.287287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.287180    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd" event={"ID":"88c0f38e-2d80-4cfb-9bc5-70dcaf27011b","Type":"ContainerDied","Data":"3da1389ec3e3f5c4306959a8f3d55812536ef4b2b1c750902979a9b89e82c18a"}
Oct 30 17:46:08.287287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.287212    5534 scope.go:122] "RemoveContainer" containerID="560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05"
Oct 30 17:46:08.288899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.288875    5534 generic.go:356] "Generic (PLEG): container finished" podID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerID="e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0" exitCode=0
Oct 30 17:46:08.289047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.288942    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"
Oct 30 17:46:08.289047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.288956    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" event={"ID":"4c9570be-a0d3-4b42-849d-9523b3f794f9","Type":"ContainerDied","Data":"e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0"}
Oct 30 17:46:08.289047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.289022    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd" event={"ID":"4c9570be-a0d3-4b42-849d-9523b3f794f9","Type":"ContainerDied","Data":"9e55b707a29b916fc57c86e769185ed7fc150a79b985084ec7f2398b4df5193d"}
Oct 30 17:46:08.301742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.301721    5534 scope.go:122] "RemoveContainer" containerID="74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"
Oct 30 17:46:08.312391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.312373    5534 scope.go:122] "RemoveContainer" containerID="560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05"
Oct 30 17:46:08.312702 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:46:08.312681    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05\": container with ID starting with 560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05 not found: ID does not exist" containerID="560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05"
Oct 30 17:46:08.312774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.312715    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05"} err="failed to get container status \"560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05\": rpc error: code = NotFound desc = could not find container \"560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05\": container with ID starting with 560cd70e865a6a0c55fe57cf3a9e141502797e884a6902d0b6ea666cbad64d05 not found: ID does not exist"
Oct 30 17:46:08.312774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.312738    5534 scope.go:122] "RemoveContainer" containerID="74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"
Oct 30 17:46:08.313032 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:46:08.313014    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239\": container with ID starting with 74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239 not found: ID does not exist" containerID="74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"
Oct 30 17:46:08.313101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.313054    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239"} err="failed to get container status \"74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239\": rpc error: code = NotFound desc = could not find container \"74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239\": container with ID starting with 74f1f7a6691e78282636be29301a03e7a9ea29cd547741e751abe2b703dc6239 not found: ID does not exist"
Oct 30 17:46:08.313101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.313076    5534 scope.go:122] "RemoveContainer" containerID="e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0"
Oct 30 17:46:08.323514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.323497    5534 scope.go:122] "RemoveContainer" containerID="e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0"
Oct 30 17:46:08.323814 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:46:08.323794    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0\": container with ID starting with e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0 not found: ID does not exist" containerID="e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0"
Oct 30 17:46:08.323907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.323818    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0"} err="failed to get container status \"e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0\": rpc error: code = NotFound desc = could not find container \"e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0\": container with ID starting with e0f0f3675778158ee1a2fc8ba5a8b0dba203c57bae0293d6d4d4857c41ed2ea0 not found: ID does not exist"
Oct 30 17:46:08.356826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.356801    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"]
Oct 30 17:46:08.375820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.375793    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-c6kpd"]
Oct 30 17:46:08.397299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.397262    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"]
Oct 30 17:46:08.400809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:08.400784    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-wxndd"]
Oct 30 17:46:09.043680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.043628    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:09.532483 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.532446    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:09.711190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.711154    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:09.777820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.777790    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"]
Oct 30 17:46:09.778509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778496    5534 cpu_manager.go:414] "containerMap: removing container" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy"
Oct 30 17:46:09.778562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778511    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy"
Oct 30 17:46:09.778562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778544    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy"
Oct 30 17:46:09.778562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778549    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy"
Oct 30 17:46:09.778708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778680    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" containerName="istio-proxy"
Oct 30 17:46:09.778708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778691    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy"
Oct 30 17:46:09.778708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.778703    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy"
Oct 30 17:46:09.783596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.783536    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:09.804350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.804308    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"]
Oct 30 17:46:09.804767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.804750    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4a271c\""
Oct 30 17:46:09.810214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.810178    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4a271c-dockercfg-x5bzz\""
Oct 30 17:46:09.824199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.824170    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"]
Oct 30 17:46:09.824925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.824906    5534 cpu_manager.go:414] "containerMap: removing container" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy"
Oct 30 17:46:09.824925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.824923    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" containerName="istio-proxy"
Oct 30 17:46:09.828335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.828316    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:09.874452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.874417    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"]
Oct 30 17:46:09.891399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.891369    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:09.891616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.891415    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:09.891616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.891561    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbscj\" (UniqueName: \"kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj\") pod \"payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:09.958725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:09.891630    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvqr\" (UniqueName: \"kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr\") pod \"payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:10.043827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.017488    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:10.138047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.137985    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbscj\" (UniqueName: \"kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj\") pod \"payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:10.140023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.139993    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvqr\" (UniqueName: \"kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr\") pod \"payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:10.140470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.140449    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:10.140629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.140614    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:10.141753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.141715    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:10.143937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.143915    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:10.168674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.165863    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbscj\" (UniqueName: \"kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj\") pod \"payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:10.177382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.177287    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvqr\" (UniqueName: \"kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr\") pod \"payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:10.372869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.372835    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9570be-a0d3-4b42-849d-9523b3f794f9" path="/var/lib/kubelet/pods/4c9570be-a0d3-4b42-849d-9523b3f794f9/volumes"
Oct 30 17:46:10.373343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.373329    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c0f38e-2d80-4cfb-9bc5-70dcaf27011b" path="/var/lib/kubelet/pods/88c0f38e-2d80-4cfb-9bc5-70dcaf27011b/volumes"
Oct 30 17:46:10.398251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.398221    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:10.458671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.453865    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:10.711704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.711680    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"]
Oct 30 17:46:10.711848 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:10.711807    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2668aea7_9fbc_43c0_beec_eb6ce300ce35.slice/crio-a0b1f468818fa9d70a3b025125b03540e1605f5014fc82672486fc120602e9b5 WatchSource:0}: Error finding container a0b1f468818fa9d70a3b025125b03540e1605f5014fc82672486fc120602e9b5: Status 404 returned error can't find the container with id a0b1f468818fa9d70a3b025125b03540e1605f5014fc82672486fc120602e9b5
Oct 30 17:46:10.806112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:10.806080    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"]
Oct 30 17:46:10.806498 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:10.806477    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1045076d_f452_4b16_95d8_d5e2e388b9e8.slice/crio-aaa42f687ba813ea95ab43422f13fb835bc71e5365db01e8ba9bb89bdb011d00 WatchSource:0}: Error finding container aaa42f687ba813ea95ab43422f13fb835bc71e5365db01e8ba9bb89bdb011d00: Status 404 returned error can't find the container with id aaa42f687ba813ea95ab43422f13fb835bc71e5365db01e8ba9bb89bdb011d00
Oct 30 17:46:11.334680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.331304    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" event={"ID":"2668aea7-9fbc-43c0-beec-eb6ce300ce35","Type":"ContainerStarted","Data":"c4bdedaec9badfc70007f908bcbe27140dc11b274c8a588769e36e233cdcbff0"}
Oct 30 17:46:11.334680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.331372    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" event={"ID":"2668aea7-9fbc-43c0-beec-eb6ce300ce35","Type":"ContainerStarted","Data":"a0b1f468818fa9d70a3b025125b03540e1605f5014fc82672486fc120602e9b5"}
Oct 30 17:46:11.334680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.331407    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:11.334680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.334146    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" event={"ID":"1045076d-f452-4b16-95d8-d5e2e388b9e8","Type":"ContainerStarted","Data":"b79a52e2e38ff51d766d97dcd324ed7c8c4ebeb448a0eb21057480a3662a6988"}
Oct 30 17:46:11.334680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.334205    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" event={"ID":"1045076d-f452-4b16-95d8-d5e2e388b9e8","Type":"ContainerStarted","Data":"aaa42f687ba813ea95ab43422f13fb835bc71e5365db01e8ba9bb89bdb011d00"}
Oct 30 17:46:11.334680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.334368    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:11.375068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.375018    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" podStartSLOduration=2.375004418 podStartE2EDuration="2.375004418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:11.37184957 +0000 UTC m=+1977.133893385" watchObservedRunningTime="2026-10-30 17:46:11.375004418 +0000 UTC m=+1977.137048231"
Oct 30 17:46:11.414480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.414418    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" podStartSLOduration=2.414399589 podStartE2EDuration="2.414399589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:11.411931278 +0000 UTC m=+1977.173975092" watchObservedRunningTime="2026-10-30 17:46:11.414399589 +0000 UTC m=+1977.176443403"
Oct 30 17:46:11.940457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:11.940418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:12.241093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.241007    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:12.492359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.492269    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"]
Oct 30 17:46:12.498265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.498243    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.516141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.516118    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-38f445-openshift-default-dockercfg-x7t6p\""
Oct 30 17:46:12.531273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.531243    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"]
Oct 30 17:46:12.580779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.580745    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.580925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.580797    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.580925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.580850    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bwg\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.581032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.580952    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.581032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.580991    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.581111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.581097    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.581158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.581129    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.581217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.581194    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.581286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.581274    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683017    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683063    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683106    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683182    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683284    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683331    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683356    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bwg\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683388    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683442    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683524    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683590    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683748    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.683833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.683782    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.684488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.684460    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.685895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.685871    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.686011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.685995    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.706291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.706268    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.706492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.706473    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bwg\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:12.721362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.721325    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:12.816005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:12.815917    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:13.154418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:13.154383    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaef6659_67fa_4a2a_9b4b_a8dfccf45f39.slice/crio-d3cf00bda70e1542c113b8d89a18f355a6af86374c141adf19d0c87fd8b2464d WatchSource:0}: Error finding container d3cf00bda70e1542c113b8d89a18f355a6af86374c141adf19d0c87fd8b2464d: Status 404 returned error can't find the container with id d3cf00bda70e1542c113b8d89a18f355a6af86374c141adf19d0c87fd8b2464d
Oct 30 17:46:13.156138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.156114    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"]
Oct 30 17:46:13.158483 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.158446    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:13.158614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.158531    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:13.158614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.158572    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:13.160584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.160561    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"]
Oct 30 17:46:13.167907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.167887    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.190814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.190779    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.190947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.190871    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-303ba5-openshift-default-dockercfg-d5wdq\""
Oct 30 17:46:13.190947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.190901    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191042    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191140    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191199    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191226    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191350    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshkb\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191467    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.191542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.191509    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.194975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.194946    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"]
Oct 30 17:46:13.215403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.215353    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:13.295812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.295773    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.296130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.296108    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.296324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.296303    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.296446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.296430    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.296700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.296677    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshkb\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.296928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.296911    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.297073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.297057    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.297212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.297194    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.297412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.297394    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.321878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.298630    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.321878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.320735    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.321878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.321032    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.321878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.321287    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.321878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.321552    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.341679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.336110    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.400309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.398023    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.449679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.449564    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshkb\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.470043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.469255    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.473235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.471968    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" event={"ID":"baef6659-67fa-4a2a-9b4b-a8dfccf45f39","Type":"ContainerStarted","Data":"4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f"}
Oct 30 17:46:13.473235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.472031    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" event={"ID":"baef6659-67fa-4a2a-9b4b-a8dfccf45f39","Type":"ContainerStarted","Data":"d3cf00bda70e1542c113b8d89a18f355a6af86374c141adf19d0c87fd8b2464d"}
Oct 30 17:46:13.498006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.497722    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:13.520970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.520917    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podStartSLOduration=1.5209023149999998 podStartE2EDuration="1.520902315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:13.516817895 +0000 UTC m=+1979.278861707" watchObservedRunningTime="2026-10-30 17:46:13.520902315 +0000 UTC m=+1979.282946127"
Oct 30 17:46:13.731200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.731122    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:13.816802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.816770    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:13.842707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.842675    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"]
Oct 30 17:46:13.843080 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:13.843056    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050346bb_c68b_4503_a695_185ed33cbf6c.slice/crio-c7c5b3e433906bc9017dfaf9f06b52feafb650e0949d6ba78efc09bf626a1462 WatchSource:0}: Error finding container c7c5b3e433906bc9017dfaf9f06b52feafb650e0949d6ba78efc09bf626a1462: Status 404 returned error can't find the container with id c7c5b3e433906bc9017dfaf9f06b52feafb650e0949d6ba78efc09bf626a1462
Oct 30 17:46:13.846812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.846775    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:13.846902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.846854    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:13.846902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.846889    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:46:13.892934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:13.892900    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:14.026736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:14.019522    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:14.503662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:14.502980    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" event={"ID":"050346bb-c68b-4503-a695-185ed33cbf6c","Type":"ContainerStarted","Data":"5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"}
Oct 30 17:46:14.503662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:14.503039    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" event={"ID":"050346bb-c68b-4503-a695-185ed33cbf6c","Type":"ContainerStarted","Data":"c7c5b3e433906bc9017dfaf9f06b52feafb650e0949d6ba78efc09bf626a1462"}
Oct 30 17:46:14.550072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:14.550009    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podStartSLOduration=1.549994469 podStartE2EDuration="1.549994469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:14.548266252 +0000 UTC m=+1980.310310067" watchObservedRunningTime="2026-10-30 17:46:14.549994469 +0000 UTC m=+1980.312038282"
Oct 30 17:46:14.818534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:14.818450    5534 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 17:46:14.818534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:14.818507    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 17:46:15.246609 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:46:15.246574    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:46:15.499779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.499604    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:15.508280 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.508252    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:15.513402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.513374    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:15.514443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.514420    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:46:15.734460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.734418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:15.818733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.818042    5534 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 17:46:15.818733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:15.818102    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 17:46:16.637409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:16.637377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:16.816922 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:16.816887    5534 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 17:46:16.817113 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:16.816943    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 17:46:16.982388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:16.982293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:17.108896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:17.108864    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:17.586164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:17.586129    5534 scope.go:122] "RemoveContainer" containerID="b325e7a41098a6185e9931daf8f24d60feb84c6e779135f9a2b2bb19d6b84268"
Oct 30 17:46:17.817366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:17.817327    5534 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 17:46:17.817988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:17.817956    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 17:46:17.986669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:17.986581    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:18.713820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.713780    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:18.817135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.817086    5534 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 17:46:18.817298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.817151    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 17:46:18.843378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.843344    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:18.896518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.896487    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"]
Oct 30 17:46:18.912887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.912859    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"]
Oct 30 17:46:18.913170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:18.913155    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.044928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.044829    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.219370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.219335    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5m4\" (UniqueName: \"kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4\") pod \"payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.220877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.047204    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-303ba5\""
Oct 30 17:46:19.227721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.049847    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-303ba5-dockercfg-2mgfs\""
Oct 30 17:46:19.320817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.320714    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.320817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.320805    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5m4\" (UniqueName: \"kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4\") pod \"payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.321572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.321537    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.344231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.344196    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5m4\" (UniqueName: \"kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4\") pod \"payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.502791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.502745    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:19.817548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.817513    5534 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 17:46:19.817769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.817573    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 17:46:19.849601 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:19.849550    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c82e6b3_5b85_4371_977d_0aa90358fac2.slice/crio-ac5b77f969f9c774d5a0ab4fe39dda0b2ef560cb93d06f9f523557f904916627 WatchSource:0}: Error finding container ac5b77f969f9c774d5a0ab4fe39dda0b2ef560cb93d06f9f523557f904916627: Status 404 returned error can't find the container with id ac5b77f969f9c774d5a0ab4fe39dda0b2ef560cb93d06f9f523557f904916627
Oct 30 17:46:19.850976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.850477    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"]
Oct 30 17:46:19.940988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:19.940959    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:20.568339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.568299    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" event={"ID":"3c82e6b3-5b85-4371-977d-0aa90358fac2","Type":"ContainerStarted","Data":"dd4eb1ca1cd9ee0171262ccd4e573cc0a121e937b7c56a2699c852c162725f45"}
Oct 30 17:46:20.568526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.568374    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:20.568526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.568401    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" event={"ID":"3c82e6b3-5b85-4371-977d-0aa90358fac2","Type":"ContainerStarted","Data":"ac5b77f969f9c774d5a0ab4fe39dda0b2ef560cb93d06f9f523557f904916627"}
Oct 30 17:46:20.609531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.609472    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" podStartSLOduration=2.609458226 podStartE2EDuration="2.609458226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:20.607772883 +0000 UTC m=+1986.369816695" watchObservedRunningTime="2026-10-30 17:46:20.609458226 +0000 UTC m=+1986.371502045"
Oct 30 17:46:20.821527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.821448    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:20.821915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.821901    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:20.822631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:20.822612    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:46:21.740545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:21.740471    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:22.346075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:22.346046    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:46:22.346288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:22.346116    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:46:23.592180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:23.592142    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:25.238164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:25.238063    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:26.909304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:26.909253    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:28.747798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:28.747748    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:30.415794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:30.415767    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:31.581995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:31.581964    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:46:32.101415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:32.101375    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:33.812343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:33.812311    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:33.938047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:33.938006    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:34.052539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:34.052506    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:34.165410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:34.165378    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:34.895605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:34.895563    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:35.518053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:35.517998    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:35.969466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:35.969430    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:36.086931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:36.086843    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:37.332785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:37.332743    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:39.511670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:39.502043    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:41.241736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:41.241699    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:43.010322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:43.010287    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:44.978119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:44.978084    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:45.246797 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:46:45.246713    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:46:46.699898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:46.699862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:48.319073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:48.319041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:49.910977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:49.910939    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:51.570412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.570368    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"]
Oct 30 17:46:51.575392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.575364    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.592544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.592505    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4a271c-serving-cert\""
Oct 30 17:46:51.598599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.598550    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"]
Oct 30 17:46:51.623904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.623863    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.624282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.624254    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.624539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.624512    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgl96\" (UniqueName: \"kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.624685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.624577    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.671404 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.671370    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:51.725954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.725913    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.726162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.726073    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.726162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.726131    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.726287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.726182    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgl96\" (UniqueName: \"kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.726769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.726740    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.728947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.728922    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.729057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.728922    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.749934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.749913    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgl96\" (UniqueName: \"kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96\") pod \"maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:51.822532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.822284    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:51.893002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:51.892969    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:52.212450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:52.212421    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"]
Oct 30 17:46:52.212587 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:46:52.212499    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2601a6f_c711_41c0_9620_f311e6ba91d5.slice/crio-49dc5e97b75264749c5ec14d45a4954c607e48e2ca57ff3f86e3945326b7485f WatchSource:0}: Error finding container 49dc5e97b75264749c5ec14d45a4954c607e48e2ca57ff3f86e3945326b7485f: Status 404 returned error can't find the container with id 49dc5e97b75264749c5ec14d45a4954c607e48e2ca57ff3f86e3945326b7485f
Oct 30 17:46:52.214010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:52.213991    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:46:52.866861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:52.866814    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh" event={"ID":"b2601a6f-c711-41c0-9620-f311e6ba91d5","Type":"ContainerStarted","Data":"1a44dad4a3a27ae0d24d16f50f31589429215f80d69197d6c1d6d4620b5f23e4"}
Oct 30 17:46:52.867356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:52.866897    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh" event={"ID":"b2601a6f-c711-41c0-9620-f311e6ba91d5","Type":"ContainerStarted","Data":"49dc5e97b75264749c5ec14d45a4954c607e48e2ca57ff3f86e3945326b7485f"}
Oct 30 17:46:52.867356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:52.866922    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:52.913787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:52.913722    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh" podStartSLOduration=1.4169015 podStartE2EDuration="1.913700042s" totalImagesPullingTime="496.798542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:52.910310508 +0000 UTC m=+2018.672354346" watchObservedRunningTime="2026-10-30 17:46:52.913700042 +0000 UTC m=+2018.675743856"
Oct 30 17:46:53.516804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:53.516776    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:55.319793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:55.319757    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:56.452254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:56.452217    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"]
Oct 30 17:46:56.452782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:56.452404    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy" containerID="cri-o://26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76" gracePeriod=30
Oct 30 17:46:56.973571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:56.973536    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:58.597537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:58.597494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:46:58.884898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:58.884811    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:46:59.093985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:59.093941    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"]
Oct 30 17:46:59.098765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:46:59.098735    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af79b3ae-4r6zc"]
Oct 30 17:47:00.373301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:00.373266    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e017f70d-c0a3-4ad3-a567-9d5576fcc3be" path="/var/lib/kubelet/pods/e017f70d-c0a3-4ad3-a567-9d5576fcc3be/volumes"
Oct 30 17:47:00.451543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:00.451496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:00.573556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:00.573517    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:01.851173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.851126    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:47:01.953973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.953934    5534 generic.go:356] "Generic (PLEG): container finished" podID="61511210-6491-4043-b94a-f24550e42e66" containerID="26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76" exitCode=0
Oct 30 17:47:01.954211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.954190    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" event={"ID":"61511210-6491-4043-b94a-f24550e42e66","Type":"ContainerDied","Data":"26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76"}
Oct 30 17:47:01.954270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.954247    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9" event={"ID":"61511210-6491-4043-b94a-f24550e42e66","Type":"ContainerDied","Data":"012aacaeb651c7a134a5efb056cccfce84023507ce904b92dce4c81b9eb428fd"}
Oct 30 17:47:01.954315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.954277    5534 scope.go:122] "RemoveContainer" containerID="26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76"
Oct 30 17:47:01.954447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.954433    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"
Oct 30 17:47:01.982664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.982609    5534 scope.go:122] "RemoveContainer" containerID="26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76"
Oct 30 17:47:01.983348 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:47:01.983272    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76\": container with ID starting with 26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76 not found: ID does not exist" containerID="26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76"
Oct 30 17:47:01.983348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:01.983314    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76"} err="failed to get container status \"26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76\": rpc error: code = NotFound desc = could not find container \"26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76\": container with ID starting with 26f044a601390f89a7c8a9bd32e05bbf2a4ac0c948b34807db39978d702e9c76 not found: ID does not exist"
Oct 30 17:47:02.006198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006164    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006234    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006267    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006306    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006339    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006427    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006467    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006499    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.006803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006562    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r\") pod \"61511210-6491-4043-b94a-f24550e42e66\" (UID: \"61511210-6491-4043-b94a-f24550e42e66\") "
Oct 30 17:47:02.007041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.006990    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:47:02.007199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007172    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:47:02.007307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007285    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:47:02.007307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007218    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:47:02.007444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007072    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:47:02.007739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007718    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61511210-6491-4043-b94a-f24550e42e66-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.007853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007742    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.007853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007756    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.007853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007770    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.007853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.007783    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.009735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.009710    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:47:02.009859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.009839    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:47:02.009954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.009934    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:47:02.009994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.009966    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r" pod "61511210-6491-4043-b94a-f24550e42e66" (UID: "61511210-6491-4043-b94a-f24550e42e66"). InnerVolumeSpecName "kube-api-access-sn64r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:47:02.112226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.111202    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn64r\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-kube-api-access-sn64r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.112226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.111245    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61511210-6491-4043-b94a-f24550e42e66-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.112226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.111260    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61511210-6491-4043-b94a-f24550e42e66-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.112226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.111273    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61511210-6491-4043-b94a-f24550e42e66-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:02.302473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.302419    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"]
Oct 30 17:47:02.309410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.309379    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:02.311290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.310767    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-glnh9"]
Oct 30 17:47:02.384840 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:02.384751    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61511210-6491-4043-b94a-f24550e42e66" path="/var/lib/kubelet/pods/61511210-6491-4043-b94a-f24550e42e66/volumes"
Oct 30 17:47:03.380104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.380062    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"]
Oct 30 17:47:03.381264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.381242    5534 cpu_manager.go:414] "containerMap: removing container" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy"
Oct 30 17:47:03.381329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.381266    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy"
Oct 30 17:47:03.381529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.381516    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="61511210-6491-4043-b94a-f24550e42e66" containerName="istio-proxy"
Oct 30 17:47:03.387060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.387036    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.404846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.404815    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"]
Oct 30 17:47:03.412286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.412237    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:47:03.412476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.412286    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:47:03.538430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.538391    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5sj\" (UniqueName: \"kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.538718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.538621    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.538718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.538695    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.640540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.640431    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5sj\" (UniqueName: \"kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.640807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.640587    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.640807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.640620    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.641305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.641277    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.643213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.643192    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.664795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.664762    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5sj\" (UniqueName: \"kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj\") pod \"maas-api-revoke-sub-fdb9053a-bh8wd\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:03.703948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:03.703905    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:04.045690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:04.045655    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"]
Oct 30 17:47:04.046005 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:47:04.045982    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fe312b_74d3_4040_af1e_d8aa8621ba47.slice/crio-18a675afad95277429d3cd5941b0b7ce237bf06c7f97963c34be3609c8915207 WatchSource:0}: Error finding container 18a675afad95277429d3cd5941b0b7ce237bf06c7f97963c34be3609c8915207: Status 404 returned error can't find the container with id 18a675afad95277429d3cd5941b0b7ce237bf06c7f97963c34be3609c8915207
Oct 30 17:47:04.208501 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:04.208462    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:04.983948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:04.983903    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd" event={"ID":"f1fe312b-74d3-4040-af1e-d8aa8621ba47","Type":"ContainerStarted","Data":"a5f95660777ff99c5b676eb66bb23f2194c54f3f61f1e3ab114a71899c2ffc26"}
Oct 30 17:47:04.984525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:04.983966    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd" event={"ID":"f1fe312b-74d3-4040-af1e-d8aa8621ba47","Type":"ContainerStarted","Data":"18a675afad95277429d3cd5941b0b7ce237bf06c7f97963c34be3609c8915207"}
Oct 30 17:47:05.027782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:05.027720    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd" podStartSLOduration=2.027699557 podStartE2EDuration="2.027699557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:05.023338862 +0000 UTC m=+2030.785382675" watchObservedRunningTime="2026-10-30 17:47:05.027699557 +0000 UTC m=+2030.789743371"
Oct 30 17:47:05.920522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:05.920492    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:05.997393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:05.997343    5534 generic.go:356] "Generic (PLEG): container finished" podID="f1fe312b-74d3-4040-af1e-d8aa8621ba47" containerID="a5f95660777ff99c5b676eb66bb23f2194c54f3f61f1e3ab114a71899c2ffc26" exitCode=0
Oct 30 17:47:05.997829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:05.997447    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd" event={"ID":"f1fe312b-74d3-4040-af1e-d8aa8621ba47","Type":"ContainerDied","Data":"a5f95660777ff99c5b676eb66bb23f2194c54f3f61f1e3ab114a71899c2ffc26"}
Oct 30 17:47:06.474220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:06.474186    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:07.210577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.210551    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:07.310196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.310113    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca\") pod \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") "
Oct 30 17:47:07.310347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.310306    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj\" (UniqueName: \"kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj\") pod \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") "
Oct 30 17:47:07.310566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.310544    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca\") pod \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\" (UID: \"f1fe312b-74d3-4040-af1e-d8aa8621ba47\") "
Oct 30 17:47:07.311219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.311193    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca" pod "f1fe312b-74d3-4040-af1e-d8aa8621ba47" (UID: "f1fe312b-74d3-4040-af1e-d8aa8621ba47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:47:07.312550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.312530    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca" pod "f1fe312b-74d3-4040-af1e-d8aa8621ba47" (UID: "f1fe312b-74d3-4040-af1e-d8aa8621ba47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:47:07.313058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.313034    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj" pod "f1fe312b-74d3-4040-af1e-d8aa8621ba47" (UID: "f1fe312b-74d3-4040-af1e-d8aa8621ba47"). InnerVolumeSpecName "kube-api-access-rp5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:47:07.412845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.412815    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:07.412845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.412840    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5sj\" (UniqueName: \"kubernetes.io/projected/f1fe312b-74d3-4040-af1e-d8aa8621ba47-kube-api-access-rp5sj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:07.412845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.412849    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe312b-74d3-4040-af1e-d8aa8621ba47-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:07.589794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:07.589767    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:08.015440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:08.015360    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"
Oct 30 17:47:08.015440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:08.015386    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd" event={"ID":"f1fe312b-74d3-4040-af1e-d8aa8621ba47","Type":"ContainerDied","Data":"18a675afad95277429d3cd5941b0b7ce237bf06c7f97963c34be3609c8915207"}
Oct 30 17:47:08.015440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:08.015432    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a675afad95277429d3cd5941b0b7ce237bf06c7f97963c34be3609c8915207"
Oct 30 17:47:09.494134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:09.494084    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:11.104520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:11.104481    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:12.847899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:12.847867    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:14.310175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.310132    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:14.800913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.800834    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:14.898470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.898434    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"]
Oct 30 17:47:14.899738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.899626    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f1fe312b-74d3-4040-af1e-d8aa8621ba47" containerName="revoke-keys"
Oct 30 17:47:14.899926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.899903    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fe312b-74d3-4040-af1e-d8aa8621ba47" containerName="revoke-keys"
Oct 30 17:47:14.900126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.900111    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fe312b-74d3-4040-af1e-d8aa8621ba47" containerName="revoke-keys"
Oct 30 17:47:14.904838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.904817    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:14.918565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.918530    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"]
Oct 30 17:47:14.926824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.925256    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:47:14.926824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.925305    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:47:14.987423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.987380    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:14.987423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.987428    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:14.987732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:14.987712    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbvb\" (UniqueName: \"kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.088886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.088781    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbvb\" (UniqueName: \"kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.089095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.088943    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.089095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.088981    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.089848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.089816    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.091794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.091769    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.112880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.112856    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbvb\" (UniqueName: \"kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb\") pod \"maas-api-revoke-sub-c4d0016b-4fmwd\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.225991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.225962    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:15.249518 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:47:15.249479    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:47:15.543388 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:47:15.543358    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffb2777_53f9_4a17_acc9_5aeb50c618ee.slice/crio-02f2918e2165fc3e255e03318ffebaec937db90e0c9622961d3fdf8af3c344d5 WatchSource:0}: Error finding container 02f2918e2165fc3e255e03318ffebaec937db90e0c9622961d3fdf8af3c344d5: Status 404 returned error can't find the container with id 02f2918e2165fc3e255e03318ffebaec937db90e0c9622961d3fdf8af3c344d5
Oct 30 17:47:15.543789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:15.543762    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"]
Oct 30 17:47:16.088631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:16.088588    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd" event={"ID":"dffb2777-53f9-4a17-acc9-5aeb50c618ee","Type":"ContainerStarted","Data":"4729bfa9bbbcfd46036b51cec3f750d50aba481fbba3af7d6ede33e91f008e24"}
Oct 30 17:47:16.088846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:16.088675    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd" event={"ID":"dffb2777-53f9-4a17-acc9-5aeb50c618ee","Type":"ContainerStarted","Data":"02f2918e2165fc3e255e03318ffebaec937db90e0c9622961d3fdf8af3c344d5"}
Oct 30 17:47:16.131381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:16.131272    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd" podStartSLOduration=2.131258044 podStartE2EDuration="2.131258044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:16.128209796 +0000 UTC m=+2041.890253609" watchObservedRunningTime="2026-10-30 17:47:16.131258044 +0000 UTC m=+2041.893301860"
Oct 30 17:47:16.153402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:16.153371    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:16.396535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:16.396438    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:16.764336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:16.764235    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:17.040842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:17.040765    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:17.100573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:17.100536    5534 generic.go:356] "Generic (PLEG): container finished" podID="dffb2777-53f9-4a17-acc9-5aeb50c618ee" containerID="4729bfa9bbbcfd46036b51cec3f750d50aba481fbba3af7d6ede33e91f008e24" exitCode=0
Oct 30 17:47:17.100812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:17.100625    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd" event={"ID":"dffb2777-53f9-4a17-acc9-5aeb50c618ee","Type":"ContainerDied","Data":"4729bfa9bbbcfd46036b51cec3f750d50aba481fbba3af7d6ede33e91f008e24"}
Oct 30 17:47:18.088576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.088496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:18.318423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.318398    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:18.462112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.462078    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca\") pod \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") "
Oct 30 17:47:18.462327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.462302    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb\" (UniqueName: \"kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb\") pod \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") "
Oct 30 17:47:18.462415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.462402    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca\") pod \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\" (UID: \"dffb2777-53f9-4a17-acc9-5aeb50c618ee\") "
Oct 30 17:47:18.462524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.462483    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca" pod "dffb2777-53f9-4a17-acc9-5aeb50c618ee" (UID: "dffb2777-53f9-4a17-acc9-5aeb50c618ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:47:18.463732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.463637    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:18.465426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.465404    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca" pod "dffb2777-53f9-4a17-acc9-5aeb50c618ee" (UID: "dffb2777-53f9-4a17-acc9-5aeb50c618ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:47:18.465542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.465452    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb" pod "dffb2777-53f9-4a17-acc9-5aeb50c618ee" (UID: "dffb2777-53f9-4a17-acc9-5aeb50c618ee"). InnerVolumeSpecName "kube-api-access-qvbvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:47:18.565097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.565059    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb2777-53f9-4a17-acc9-5aeb50c618ee-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:18.565097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:18.565092    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbvb\" (UniqueName: \"kubernetes.io/projected/dffb2777-53f9-4a17-acc9-5aeb50c618ee-kube-api-access-qvbvb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:19.118929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:19.118897    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"
Oct 30 17:47:19.119410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:19.118925    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd" event={"ID":"dffb2777-53f9-4a17-acc9-5aeb50c618ee","Type":"ContainerDied","Data":"02f2918e2165fc3e255e03318ffebaec937db90e0c9622961d3fdf8af3c344d5"}
Oct 30 17:47:19.119410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:19.118972    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f2918e2165fc3e255e03318ffebaec937db90e0c9622961d3fdf8af3c344d5"
Oct 30 17:47:19.943188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:19.943142    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:21.765724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:21.765686    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:23.531303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:23.531268    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:25.303104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:25.303056    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:27.175852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:27.175806    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:29.276667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:29.276182    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:31.843389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:31.843319    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:32.387667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:32.386793    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:32.880213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:32.880161    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:33.717613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:33.433019    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:33.867220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:33.867184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:34.343458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:34.343423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:34.554385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:34.554340    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 17:47:34.645925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:34.645809    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:34.680358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:34.680288    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx" podUID="e3ec1506-9ff5-48f6-9bd7-d19a6a095892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:34.726708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:34.725773    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:34.753754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:34.747745    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:36.444798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:36.444760    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:36.931222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:36.931145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:37.060548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:37.060512    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:37.309875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:37.309788    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:47:37.310040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:37.309876    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:38.181683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:38.181621    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:38.493015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:38.492926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:38.681904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:38.681864    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"]
Oct 30 17:47:38.683458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:38.683432    5534 cpu_manager.go:414] "containerMap: removing container" podUID="dffb2777-53f9-4a17-acc9-5aeb50c618ee" containerName="revoke-keys"
Oct 30 17:47:38.683631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:38.683617    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffb2777-53f9-4a17-acc9-5aeb50c618ee" containerName="revoke-keys"
Oct 30 17:47:38.683941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:38.683927    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffb2777-53f9-4a17-acc9-5aeb50c618ee" containerName="revoke-keys"
Oct 30 17:47:39.127164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.127131    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"]
Oct 30 17:47:39.127354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.127231    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.141500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.141478    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:47:39.141786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.141772    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:47:39.199847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.199815    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44f4\" (UniqueName: \"kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.200332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.200065    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.200332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.200119    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.302202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.302167    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44f4\" (UniqueName: \"kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.302405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.302353    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.302405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.302395    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.303348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.303319    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.305305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.305279    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.325567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.325542    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44f4\" (UniqueName: \"kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4\") pod \"maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.493746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.493637    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:39.798809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.798771    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:47:39.798960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:39.798825    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:40.272549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:40.272517    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:40.384691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:40.384628    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:40.671663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:40.640762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:40.820808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:40.820769    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"]
Oct 30 17:47:41.021546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.020918    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"]
Oct 30 17:47:41.021546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.021034    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.021546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.021193    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:41.050979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.050956    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bcdac0\""
Oct 30 17:47:41.051158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.051143    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bcdac0-dockercfg-hh44s\""
Oct 30 17:47:41.172533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.172493    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zd7t\" (UniqueName: \"kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t\") pod \"payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.172778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.172626    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.200495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.200456    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:41.274133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.274042    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zd7t\" (UniqueName: \"kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t\") pod \"payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.274763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.274737    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.275929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.275804    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.340053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.340026    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zd7t\" (UniqueName: \"kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t\") pod \"payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.605033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.604997    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:41.681713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.681618    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"]
Oct 30 17:47:41.865553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:41.865411    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt" event={"ID":"184fe94d-def3-4f4a-a883-7da17dd7301d","Type":"ContainerStarted","Data":"66166327f7a1ec376d8190b6146834391e8d3817547e532b7c3ee8e4595134af"}
Oct 30 17:47:42.006526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.006466    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"]
Oct 30 17:47:42.013932 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:47:42.013864    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759e7c9d_045a_4b10_8ffb_c5773e6e3334.slice/crio-324826724f4deda751bcca68010c78541ae1cb01c3e7a36492615d5c137d71d4 WatchSource:0}: Error finding container 324826724f4deda751bcca68010c78541ae1cb01c3e7a36492615d5c137d71d4: Status 404 returned error can't find the container with id 324826724f4deda751bcca68010c78541ae1cb01c3e7a36492615d5c137d71d4
Oct 30 17:47:42.670074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.670042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:42.879497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.879406    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc" event={"ID":"759e7c9d-045a-4b10-8ffb-c5773e6e3334","Type":"ContainerStarted","Data":"95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808"}
Oct 30 17:47:42.879497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.879470    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc" event={"ID":"759e7c9d-045a-4b10-8ffb-c5773e6e3334","Type":"ContainerStarted","Data":"324826724f4deda751bcca68010c78541ae1cb01c3e7a36492615d5c137d71d4"}
Oct 30 17:47:42.879838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.879624    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:42.882271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.882241    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt" event={"ID":"184fe94d-def3-4f4a-a883-7da17dd7301d","Type":"ContainerStarted","Data":"b05a7dcab402648c015fff2369d22605291dc74e60e1c894e09a3eeae603a717"}
Oct 30 17:47:42.963944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:42.963879    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc" podStartSLOduration=2.9638532680000003 podStartE2EDuration="2.963853268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:42.957727238 +0000 UTC m=+2068.719771049" watchObservedRunningTime="2026-10-30 17:47:42.963853268 +0000 UTC m=+2068.725897074"
Oct 30 17:47:43.005623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.005565    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt" podStartSLOduration=5.005551248 podStartE2EDuration="5.005551248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:43.001970486 +0000 UTC m=+2068.764014315" watchObservedRunningTime="2026-10-30 17:47:43.005551248 +0000 UTC m=+2068.767595062"
Oct 30 17:47:43.485568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.485528    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"]
Oct 30 17:47:43.542894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.542814    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"]
Oct 30 17:47:43.543192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.543163    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647405    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647477    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647582    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647624    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68n8l\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647673    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647813    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647955    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.647999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.647984    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.749723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.749613    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.749753    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.749804    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.749831    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.749933    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750003    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750229    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750288    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750309    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68n8l\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750347    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750690    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.750952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750907    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.751081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.750981    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.751081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.751063    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.752400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.752379    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.763426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.763326    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.774194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.774139    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.774345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.774274    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68n8l\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.900182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.898586    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:43.900182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.899160    5534 generic.go:356] "Generic (PLEG): container finished" podID="184fe94d-def3-4f4a-a883-7da17dd7301d" containerID="b05a7dcab402648c015fff2369d22605291dc74e60e1c894e09a3eeae603a717" exitCode=0
Oct 30 17:47:43.900182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:43.899374    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt" event={"ID":"184fe94d-def3-4f4a-a883-7da17dd7301d","Type":"ContainerDied","Data":"b05a7dcab402648c015fff2369d22605291dc74e60e1c894e09a3eeae603a717"}
Oct 30 17:47:44.387438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.387400    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"]
Oct 30 17:47:44.387752 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:47:44.387718    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122c81ef_c322_48bb_9a02_45fd6359b6cd.slice/crio-49484e620ed4fadbc72e60ddc96b166fb327e0efc59ab1943d0ffde4193a202d WatchSource:0}: Error finding container 49484e620ed4fadbc72e60ddc96b166fb327e0efc59ab1943d0ffde4193a202d: Status 404 returned error can't find the container with id 49484e620ed4fadbc72e60ddc96b166fb327e0efc59ab1943d0ffde4193a202d
Oct 30 17:47:44.391622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.391585    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:44.391739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.391696    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:44.391789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.391741    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:44.553006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.552978    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:44.855129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.855097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:44.935152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.935117    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" event={"ID":"122c81ef-c322-48bb-9a02-45fd6359b6cd","Type":"ContainerStarted","Data":"71fc1b9c2dd52dbbf5337ed16c73eb7436e3b53b16877e8e0f92d985b80b696a"}
Oct 30 17:47:44.935351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:44.935178    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" event={"ID":"122c81ef-c322-48bb-9a02-45fd6359b6cd","Type":"ContainerStarted","Data":"49484e620ed4fadbc72e60ddc96b166fb327e0efc59ab1943d0ffde4193a202d"}
Oct 30 17:47:45.378221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.330738    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:45.391671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.386930    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:45.425326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.425241    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podStartSLOduration=2.4252199660000002 podStartE2EDuration="2.425219966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:44.984920728 +0000 UTC m=+2070.746964541" watchObservedRunningTime="2026-10-30 17:47:45.425219966 +0000 UTC m=+2071.187263780"
Oct 30 17:47:45.508419 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.508293    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4\" (UniqueName: \"kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4\") pod \"184fe94d-def3-4f4a-a883-7da17dd7301d\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") "
Oct 30 17:47:45.508592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.508441    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca\") pod \"184fe94d-def3-4f4a-a883-7da17dd7301d\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") "
Oct 30 17:47:45.508592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.508478    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca\") pod \"184fe94d-def3-4f4a-a883-7da17dd7301d\" (UID: \"184fe94d-def3-4f4a-a883-7da17dd7301d\") "
Oct 30 17:47:45.509135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.509088    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca" pod "184fe94d-def3-4f4a-a883-7da17dd7301d" (UID: "184fe94d-def3-4f4a-a883-7da17dd7301d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:47:45.525166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.524066    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4" pod "184fe94d-def3-4f4a-a883-7da17dd7301d" (UID: "184fe94d-def3-4f4a-a883-7da17dd7301d"). InnerVolumeSpecName "kube-api-access-h44f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:47:45.526964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.526919    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca" pod "184fe94d-def3-4f4a-a883-7da17dd7301d" (UID: "184fe94d-def3-4f4a-a883-7da17dd7301d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:47:45.537912 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:47:45.537863    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:47:45.619884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.619846    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h44f4\" (UniqueName: \"kubernetes.io/projected/184fe94d-def3-4f4a-a883-7da17dd7301d-kube-api-access-h44f4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:45.619884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.619881    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:45.619884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.619895    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184fe94d-def3-4f4a-a883-7da17dd7301d-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:47:45.901592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.899782    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:45.908137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.907777    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 17:47:45.908137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.907854    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 17:47:45.915897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.915835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:45.974071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.974043    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"
Oct 30 17:47:45.976412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.974480    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt" event={"ID":"184fe94d-def3-4f4a-a883-7da17dd7301d","Type":"ContainerDied","Data":"66166327f7a1ec376d8190b6146834391e8d3817547e532b7c3ee8e4595134af"}
Oct 30 17:47:45.976412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:45.976284    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66166327f7a1ec376d8190b6146834391e8d3817547e532b7c3ee8e4595134af"
Oct 30 17:47:46.373667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:46.361511    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:46.563717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:46.560812    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:47:46.563717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:46.560907    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:46.900070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:46.899360    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 17:47:46.900070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:46.899420    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 17:47:47.193790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:47.165515    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:47.902842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:47.902803    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 17:47:47.903115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:47.902876    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 17:47:48.905292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:48.904838    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 17:47:48.905292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:48.904942    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 17:47:49.305969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.305880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:49.509666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.509350    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"]
Oct 30 17:47:49.542521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.542484    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"]
Oct 30 17:47:49.704036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.684560    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:49.822135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.820919    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:49.902691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.900596    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 17:47:49.902691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:49.900671    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 17:47:50.206537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:50.206447    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:50.327497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:50.325293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:50.899103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:50.899046    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 17:47:50.899301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:50.899110    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 17:47:51.644412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.644382    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"]
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.646380    5534 cpu_manager.go:414] "containerMap: removing container" podUID="184fe94d-def3-4f4a-a883-7da17dd7301d" containerName="revoke-keys"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.646402    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184fe94d-def3-4f4a-a883-7da17dd7301d" containerName="revoke-keys"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.646690    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="184fe94d-def3-4f4a-a883-7da17dd7301d" containerName="revoke-keys"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.675384    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.697700    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"]
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.797707    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.797772    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.797938    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.798027    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.798075    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.798097    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.798132    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wng\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.798293    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.798406    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.841122    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.900808    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.900864    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.900976    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901032    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901072    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901101    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901151    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wng\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901256    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901365    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.901847    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.902721    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.902940    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.903443    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.904083    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.913855    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.914090    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.915369    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.916457    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.925354    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.943831    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wng\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.945480    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-dpkj6\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:51.978492    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.066242    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.108795    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.323243    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:47:52.338135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.323299    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:47:52.765207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.765179    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"]
Oct 30 17:47:52.766855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.766820    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:52.766970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.766893    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:52.766970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:52.766936    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:53.101667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.101565    5534 generic.go:356] "Generic (PLEG): container finished" podID="42639ec9-312c-49fa-b750-3cead79a0534" containerID="737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d" exitCode=0
Oct 30 17:47:53.101882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.101766    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" event={"ID":"42639ec9-312c-49fa-b750-3cead79a0534","Type":"ContainerDied","Data":"737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"}
Oct 30 17:47:53.105778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.105729    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" event={"ID":"a694412f-e18c-4fae-99a6-e525d1cc837d","Type":"ContainerStarted","Data":"72f17919c22373d3bb79038cbde16082677b4e1b97f524ccdf040159a35c5e80"}
Oct 30 17:47:53.105914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.105802    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" event={"ID":"a694412f-e18c-4fae-99a6-e525d1cc837d","Type":"ContainerStarted","Data":"2e5bfe3405970cee2933426642f923eb2f38c63f953546a230676304a79eb602"}
Oct 30 17:47:53.113082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.113057    5534 scope.go:122] "RemoveContainer" containerID="737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"
Oct 30 17:47:53.115389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.115355    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:53.115524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.115469    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:53.115524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.115514    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:53.239826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.239102    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podStartSLOduration=2.239080158 podStartE2EDuration="2.239080158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:53.216134785 +0000 UTC m=+2078.978178600" watchObservedRunningTime="2026-10-30 17:47:53.239080158 +0000 UTC m=+2079.001123971"
Oct 30 17:47:53.925794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:53.925746    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:47:54.129667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.118827    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:47:54.129667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.124372    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 17:47:54.129667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.124428    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 17:47:54.129667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.126269    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:54.291214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.139261    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" event={"ID":"42639ec9-312c-49fa-b750-3cead79a0534","Type":"ContainerStarted","Data":"8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0"}
Oct 30 17:47:54.490787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.467216    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:54.490787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.475918    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": read tcp 10.133.0.2:37322->10.133.0.209:15021: read: connection reset by peer" start-of-body=
Oct 30 17:47:54.490787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.475977    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": read tcp 10.133.0.2:37322->10.133.0.209:15021: read: connection reset by peer"
Oct 30 17:47:54.512692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.510322    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerName="payload-processing" containerID="cri-o://340eb9b2c980441cd0142e4f382f0f3ed120278fd41134219b4d5ffa4390b2ea" gracePeriod=25
Oct 30 17:47:54.544103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.544014    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerName="payload-pre-processing" containerID="cri-o://8bcfcae8f4215a3542535fd8a211ab92eeb5553cf100bc7ab3246fa909364af0" gracePeriod=25
Oct 30 17:47:54.947669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.946465    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:47:54.947669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.946575    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 17:47:54.947669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:54.946626    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 17:47:55.081669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.067413    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 17:47:55.081669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.067477    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 17:47:55.150599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.150566    5534 generic.go:356] "Generic (PLEG): container finished" podID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerID="4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da" exitCode=0
Oct 30 17:47:55.150819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.150670    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" event={"ID":"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f","Type":"ContainerDied","Data":"4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"}
Oct 30 17:47:55.151396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.151377    5534 scope.go:122] "RemoveContainer" containerID="4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"
Oct 30 17:47:55.159920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.153691    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:55.159920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.153779    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:55.159920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.153829    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:55.947991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.947470    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 17:47:55.947991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:55.947552    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 17:47:56.072572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.072096    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 17:47:56.072572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.072161    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 17:47:56.192621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.192580    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" event={"ID":"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f","Type":"ContainerStarted","Data":"b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8"}
Oct 30 17:47:56.507386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.467949    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:56.643925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.643894    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:47:56.644111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.643994    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:47:56.644111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.644031    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:47:56.713275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.710338    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:47:56.949527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.949108    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 17:47:56.949527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:56.949166    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 17:47:57.075198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:57.075163    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 17:47:57.075551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:57.075510    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 17:47:57.647675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:57.644819    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:47:57.647675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:57.644879    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:47:57.713742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:57.710882    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:47:58.082729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.051202    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 17:47:58.082729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.051264    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 17:47:58.124516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.124388    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 17:47:58.124516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.124447    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 17:47:58.654677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.646227    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:47:58.654677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.646304    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:47:58.834577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.834502    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:47:58.959374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.959282    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 17:47:58.964242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:58.964204    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 17:47:59.082668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.079036    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 17:47:59.082668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.079107    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 17:47:59.344862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.344829    5534 generic.go:356] "Generic (PLEG): container finished" podID="050346bb-c68b-4503-a695-185ed33cbf6c" containerID="5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571" exitCode=0
Oct 30 17:47:59.345068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.344901    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" event={"ID":"050346bb-c68b-4503-a695-185ed33cbf6c","Type":"ContainerDied","Data":"5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"}
Oct 30 17:47:59.345367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.345351    5534 scope.go:122] "RemoveContainer" containerID="5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"
Oct 30 17:47:59.347282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.347245    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:59.347409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.347323    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:59.347409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.347371    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:47:59.645696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.645265    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:47:59.645696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:47:59.645320    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:48:00.007735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.004055    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 17:48:00.007735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.004140    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 17:48:00.130106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.129000    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:48:00.149372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.149342    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:48:00.248967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.248932    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:48:00.476665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.476297    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" event={"ID":"050346bb-c68b-4503-a695-185ed33cbf6c","Type":"ContainerStarted","Data":"382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e"}
Oct 30 17:48:00.507174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.499756    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:48:00.517161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.517132    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:48:00.517460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.517438    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:00.517605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.517581    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:00.646663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.643850    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:48:00.646663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.643899    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:48:00.974668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.968035    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:48:00.974668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.969260    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:48:00.984671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:00.977938    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:48:01.100665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:01.079801    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:01.513862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:01.513795    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:01.514368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:01.513860    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:01.643843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:01.643450    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:48:01.643843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:01.643499    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:48:02.499054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:02.499017    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:02.499360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:02.499333    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:02.643662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:02.643603    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bczqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:48:02.644188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:02.643703    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:48:02.978694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:02.973606    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:03.499666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:03.499619    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:03.499893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:03.499697    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:03.649135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:03.649098    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:48:03.649634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:03.649620    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:48:03.650367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:03.650344    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:48:04.499478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:04.499442    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:04.499725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:04.499514    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:04.710887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:04.710851    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:05.498917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:05.498877    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:05.499142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:05.498952    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:06.499923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.499879    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 17:48:06.500389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.499959    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 17:48:06.587262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.587229    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:06.673481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.673444    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"]
Oct 30 17:48:06.678240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.678174    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:48:06.685044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.685017    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.690236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.688072    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"]
Oct 30 17:48:06.708428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.708386    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.712092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.712016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.719769    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.719897    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.719947    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.719997    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.720046    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.720079    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.720129    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76dk\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.720161    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.722707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.720230    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.734885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.734853    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"]
Oct 30 17:48:06.737122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.737082    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"]
Oct 30 17:48:06.821341 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821310    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.821572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821354    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.821572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821377    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.821572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821404    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.821572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821426    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.821892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821571    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsj7\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.821892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821628    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76dk\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.821892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821709    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.821892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821751    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.821892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821809    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821922    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821939    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821950    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.821974    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822036    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822105    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.822200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822131    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.822616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822385    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.822616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822501    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.822616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822530    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.822616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822501    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.822966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.822944    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.823366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.823342    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.824302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.824283    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.824302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.824294    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.845131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.845111    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.845405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.845384    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76dk\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-g2fk5\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:06.923449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923414    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923458    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923488    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923682    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923768    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923832    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsj7\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923891    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.923935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923920    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.924065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.923962    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.924065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.924041    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.924201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.924072    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.924201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.924158    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.924325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.924307    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.924399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.924383    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.926416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.926389    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.926731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.926709    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.947899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.947877    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsj7\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:06.948322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:06.948301    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5bm5m\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:07.050210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.050132    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:07.064938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.064908    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:07.443018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.442961    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:48:07.443155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.443124    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:48:07.443228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.443209    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:48:07.445590 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:48:07.445560    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d71c27_0e87_45dd_8aa1_1d29ee044d5b.slice/crio-05abc52fc884a082814defdc48a7b1d901f87da84dcba51c5929b7aabc58f9bf WatchSource:0}: Error finding container 05abc52fc884a082814defdc48a7b1d901f87da84dcba51c5929b7aabc58f9bf: Status 404 returned error can't find the container with id 05abc52fc884a082814defdc48a7b1d901f87da84dcba51c5929b7aabc58f9bf
Oct 30 17:48:07.450025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.448205    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"]
Oct 30 17:48:07.452117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.450514    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:48:07.452117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.450665    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:48:07.452117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.450760    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:48:07.472536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.472506    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"]
Oct 30 17:48:07.507746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.507538    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:48:07.513858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.513838    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:48:07.568270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.568171    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" event={"ID":"7b22e37f-37d9-4687-a94c-b63c842d917a","Type":"ContainerStarted","Data":"331e6afe6ac25707e857e8a320dcd777371d616cf922e7c54ede1fd53403196d"}
Oct 30 17:48:07.568270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.568234    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" event={"ID":"7b22e37f-37d9-4687-a94c-b63c842d917a","Type":"ContainerStarted","Data":"c5a4c23f2c12be7eccbf0aa851a8cd86b7f1bf6809525885e65d2ef526a0af91"}
Oct 30 17:48:07.570530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.570502    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" event={"ID":"71d71c27-0e87-45dd-8aa1-1d29ee044d5b","Type":"ContainerStarted","Data":"128017967c2369d7f5763f20a10afcce20e34dfdb96a89d2ee0d86e9f74189b4"}
Oct 30 17:48:07.570795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.570759    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" event={"ID":"71d71c27-0e87-45dd-8aa1-1d29ee044d5b","Type":"ContainerStarted","Data":"05abc52fc884a082814defdc48a7b1d901f87da84dcba51c5929b7aabc58f9bf"}
Oct 30 17:48:07.734451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.733602    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podStartSLOduration=1.733583797 podStartE2EDuration="1.733583797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:07.680511649 +0000 UTC m=+2093.442555466" watchObservedRunningTime="2026-10-30 17:48:07.733583797 +0000 UTC m=+2093.495627610"
Oct 30 17:48:07.735994 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.735614    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:48:07.738698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:07.737934    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podStartSLOduration=1.7379223860000002 podStartE2EDuration="1.737922386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:07.733089939 +0000 UTC m=+2093.495133752" watchObservedRunningTime="2026-10-30 17:48:07.737922386 +0000 UTC m=+2093.499966199"
Oct 30 17:48:08.050902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.050812    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:08.052408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.052381    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:08.052566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.052441    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:08.065973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.065944    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:08.067787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.067763    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:08.067910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.067822    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:08.429510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:08.429478    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:09.051064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:09.051023    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:09.051546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:09.051515    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:09.075678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:09.074079    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:09.075678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:09.074139    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:10.051134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:10.051098    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:10.051533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:10.051152    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:10.075669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:10.069017    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:10.075669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:10.069077    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:10.297273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:10.297239    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:11.050436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:11.050403    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:11.050622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:11.050450    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:11.068571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:11.068538    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:11.069193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:11.069165    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:11.690131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:11.690096    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:12.055216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:12.052326    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:12.055216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:12.052388    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:12.072053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:12.072016    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:12.074153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:12.074114    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:12.537682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:12.537632    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:13.051451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:13.051421    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:13.051711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:13.051690    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:13.065981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:13.065946    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:13.066220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:13.066193    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:14.051173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.051131    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 17:48:14.051598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.051202    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 17:48:14.065599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.065560    5534 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 17:48:14.065812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.065623    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 17:48:14.537402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.537369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:14.808130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.808093    5534 generic.go:356] "Generic (PLEG): container finished" podID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerID="8bcfcae8f4215a3542535fd8a211ab92eeb5553cf100bc7ab3246fa909364af0" exitCode=0
Oct 30 17:48:14.812405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.808474    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" event={"ID":"26846168-75c8-4ec1-a92d-cfaec09a86a6","Type":"ContainerDied","Data":"8bcfcae8f4215a3542535fd8a211ab92eeb5553cf100bc7ab3246fa909364af0"}
Oct 30 17:48:14.815705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.815677    5534 generic.go:356] "Generic (PLEG): container finished" podID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerID="340eb9b2c980441cd0142e4f382f0f3ed120278fd41134219b4d5ffa4390b2ea" exitCode=0
Oct 30 17:48:14.816811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.815884    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" event={"ID":"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd","Type":"ContainerDied","Data":"340eb9b2c980441cd0142e4f382f0f3ed120278fd41134219b4d5ffa4390b2ea"}
Oct 30 17:48:14.881899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.881871    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:48:14.935567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.935538    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:48:14.959818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.959787    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh\" (UniqueName: \"kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh\") pod \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") "
Oct 30 17:48:14.959988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.959879    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb\" (UniqueName: \"kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb\") pod \"26846168-75c8-4ec1-a92d-cfaec09a86a6\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") "
Oct 30 17:48:14.960061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.960041    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume\") pod \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\" (UID: \"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd\") "
Oct 30 17:48:14.960225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.960205    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume\") pod \"26846168-75c8-4ec1-a92d-cfaec09a86a6\" (UID: \"26846168-75c8-4ec1-a92d-cfaec09a86a6\") "
Oct 30 17:48:14.961003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.960820    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume" pod "26846168-75c8-4ec1-a92d-cfaec09a86a6" (UID: "26846168-75c8-4ec1-a92d-cfaec09a86a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:48:14.961003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.960973    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume" pod "68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" (UID: "68a4a0e3-162a-4e61-a7dd-e6e7d54350dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:48:14.961187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.961152    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:14.961187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.961173    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846168-75c8-4ec1-a92d-cfaec09a86a6-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:14.962311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.962283    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh" pod "68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" (UID: "68a4a0e3-162a-4e61-a7dd-e6e7d54350dd"). InnerVolumeSpecName "kube-api-access-q42sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:14.963396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:14.963377    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb" pod "26846168-75c8-4ec1-a92d-cfaec09a86a6" (UID: "26846168-75c8-4ec1-a92d-cfaec09a86a6"). InnerVolumeSpecName "kube-api-access-df5nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:15.056799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.056712    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:15.057275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.057095    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:15.057965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.057948    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:48:15.063082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.063059    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42sh\" (UniqueName: \"kubernetes.io/projected/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd-kube-api-access-q42sh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:15.063205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.063084    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df5nb\" (UniqueName: \"kubernetes.io/projected/26846168-75c8-4ec1-a92d-cfaec09a86a6-kube-api-access-df5nb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:15.070437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.070415    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:15.070693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.070676    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:15.071357 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.071341    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:48:15.272176 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:48:15.272115    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:48:15.826161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.826068    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"
Oct 30 17:48:15.826161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.826066    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl" event={"ID":"26846168-75c8-4ec1-a92d-cfaec09a86a6","Type":"ContainerDied","Data":"bcdeba32f8910152348385c8d780aacfda56068b58e220b0ee67cb25f55d7874"}
Oct 30 17:48:15.826431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.826167    5534 scope.go:122] "RemoveContainer" containerID="8bcfcae8f4215a3542535fd8a211ab92eeb5553cf100bc7ab3246fa909364af0"
Oct 30 17:48:15.827813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.827790    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"
Oct 30 17:48:15.827960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.827814    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9" event={"ID":"68a4a0e3-162a-4e61-a7dd-e6e7d54350dd","Type":"ContainerDied","Data":"237dde49d6093c0f2c8f9d6c2d553d56e072ea01645a26f0e13751d5832f5f2f"}
Oct 30 17:48:15.838783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.838726    5534 scope.go:122] "RemoveContainer" containerID="340eb9b2c980441cd0142e4f382f0f3ed120278fd41134219b4d5ffa4390b2ea"
Oct 30 17:48:15.879479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.879449    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"]
Oct 30 17:48:15.890830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.890797    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bba90b-c87fdd447-h6kjl"]
Oct 30 17:48:15.917156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.917118    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"]
Oct 30 17:48:15.920849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:15.920825    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bba90b-96668d844-h8pb9"]
Oct 30 17:48:16.067167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.067131    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:16.527606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.527556    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" path="/var/lib/kubelet/pods/26846168-75c8-4ec1-a92d-cfaec09a86a6/volumes"
Oct 30 17:48:16.528301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.528281    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" path="/var/lib/kubelet/pods/68a4a0e3-162a-4e61-a7dd-e6e7d54350dd/volumes"
Oct 30 17:48:16.626198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.625698    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:16.740249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.740210    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:16.790316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.790213    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"]
Oct 30 17:48:16.794620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.794583    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bba90b-wmtxt"]
Oct 30 17:48:16.883803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:16.883771    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:17.204775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:17.204745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:17.325914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:17.325885    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:17.824390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:17.824362    5534 scope.go:122] "RemoveContainer" containerID="17b5fcbda415327facd505d370e61e929140d1b6f2b868d44aa6b888c2a62f92"
Oct 30 17:48:17.841659 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:17.837876    5534 scope.go:122] "RemoveContainer" containerID="9b43f3e6e7a1feea010602113a34fc62475aa3974cf5fff0e4afdc66c32f2dfa"
Oct 30 17:48:17.850959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:17.850938    5534 scope.go:122] "RemoveContainer" containerID="5d732808a209c23ace4b4acfc62d13ab94ca79b5cfa8be3ae7dca9a81f538677"
Oct 30 17:48:17.862155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:17.862135    5534 scope.go:122] "RemoveContainer" containerID="93528094050fb9782d0522e939e105162542c6721cf490cafdcd58ee52608c47"
Oct 30 17:48:18.376349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:18.376313    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184fe94d-def3-4f4a-a883-7da17dd7301d" path="/var/lib/kubelet/pods/184fe94d-def3-4f4a-a883-7da17dd7301d/volumes"
Oct 30 17:48:18.559294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:18.559264    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:18.696540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:18.696450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:20.409900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:20.409865    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:22.261528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:22.261488    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:23.954781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:23.954747    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:24.284008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:24.283907    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:24.411310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:24.411270    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:24.705988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:24.705953    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:24.711920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:24.711895    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"]
Oct 30 17:48:24.712181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:24.712145    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy" containerID="cri-o://a05a16fde58f72fbf16e5e37d8240d09f176bbc1b021ca54fd85da89c44d30dc" gracePeriod=30
Oct 30 17:48:25.222520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.222475    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"]
Oct 30 17:48:25.223479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.223457    5534 cpu_manager.go:414] "containerMap: removing container" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerName="payload-pre-processing"
Oct 30 17:48:25.223479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.223480    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerName="payload-pre-processing"
Oct 30 17:48:25.223727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.223503    5534 cpu_manager.go:414] "containerMap: removing container" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerName="payload-processing"
Oct 30 17:48:25.223727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.223510    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerName="payload-processing"
Oct 30 17:48:25.223727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.223697    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="26846168-75c8-4ec1-a92d-cfaec09a86a6" containerName="payload-pre-processing"
Oct 30 17:48:25.223727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.223712    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a4a0e3-162a-4e61-a7dd-e6e7d54350dd" containerName="payload-processing"
Oct 30 17:48:25.230008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.229987    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.248004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.247979    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"]
Oct 30 17:48:25.299238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.299201    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.299477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.299368    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.299570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.299535    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njb7r\" (UniqueName: \"kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.335508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.333785    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:48:25.335508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.333943    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:48:25.401674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.401602    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.401881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.401744    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.401956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.401911    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njb7r\" (UniqueName: \"kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.402832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.402797    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.404835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.404809    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.424704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.424681    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njb7r\" (UniqueName: \"kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r\") pod \"maas-api-revoke-keys-e2e-inf-b-143041-b9sh6\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.549403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.549336    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:25.919273 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:48:25.919246    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd025f59e_4a71_4801_bb4e_198e97a2c5ff.slice/crio-7721fa353fa9b3e5801f96a4fc86c0499f82d3905af789104d1193fb6e69be4d WatchSource:0}: Error finding container 7721fa353fa9b3e5801f96a4fc86c0499f82d3905af789104d1193fb6e69be4d: Status 404 returned error can't find the container with id 7721fa353fa9b3e5801f96a4fc86c0499f82d3905af789104d1193fb6e69be4d
Oct 30 17:48:25.919522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.919501    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"]
Oct 30 17:48:25.922555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.922496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:25.997585 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:25.997547    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6" event={"ID":"d025f59e-4a71-4801-bb4e-198e97a2c5ff","Type":"ContainerStarted","Data":"7721fa353fa9b3e5801f96a4fc86c0499f82d3905af789104d1193fb6e69be4d"}
Oct 30 17:48:27.009622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.009591    5534 generic.go:356] "Generic (PLEG): container finished" podID="d025f59e-4a71-4801-bb4e-198e97a2c5ff" containerID="4aa23c91887f5826098462856d60633b9bc0ef56a8bcfd09503bc2010638ed3c" exitCode=0
Oct 30 17:48:27.010063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.009670    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6" event={"ID":"d025f59e-4a71-4801-bb4e-198e97a2c5ff","Type":"ContainerDied","Data":"4aa23c91887f5826098462856d60633b9bc0ef56a8bcfd09503bc2010638ed3c"}
Oct 30 17:48:27.138999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.138965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:27.587710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.587660    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"]
Oct 30 17:48:27.592722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.592701    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.614268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.614241    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"]
Oct 30 17:48:27.619792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.619765    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bcdac0-serving-cert\""
Oct 30 17:48:27.675492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.675458    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:27.741615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.741582    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.741833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.741686    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.741833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.741713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbnz\" (UniqueName: \"kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.741935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.741829    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.785474 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.785442    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:27.852678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.845730    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.852678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.845832    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.852678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.845864    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbnz\" (UniqueName: \"kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.852678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.845962    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.852678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.847845    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.852678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.850961    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.878823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.874081    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.878823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.876851    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbnz\" (UniqueName: \"kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz\") pod \"maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:27.908920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.908884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:27.909267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:27.909237    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:28.244830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.244803    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:28.248995 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:48:28.248954    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6d9c6_5bf7_446d_9261_ae9d8672aa2d.slice/crio-0d86d00127260c44350b5a4ee00dbcdb4cb028863652af3f2c0c737e891dd002 WatchSource:0}: Error finding container 0d86d00127260c44350b5a4ee00dbcdb4cb028863652af3f2c0c737e891dd002: Status 404 returned error can't find the container with id 0d86d00127260c44350b5a4ee00dbcdb4cb028863652af3f2c0c737e891dd002
Oct 30 17:48:28.250368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.250343    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"]
Oct 30 17:48:28.355271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.355244    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca\") pod \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") "
Oct 30 17:48:28.355475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.355341    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca\") pod \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") "
Oct 30 17:48:28.355475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.355369    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r\" (UniqueName: \"kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r\") pod \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\" (UID: \"d025f59e-4a71-4801-bb4e-198e97a2c5ff\") "
Oct 30 17:48:28.355693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.355667    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca" pod "d025f59e-4a71-4801-bb4e-198e97a2c5ff" (UID: "d025f59e-4a71-4801-bb4e-198e97a2c5ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:48:28.356233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.356154    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:28.358133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.358103    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r" pod "d025f59e-4a71-4801-bb4e-198e97a2c5ff" (UID: "d025f59e-4a71-4801-bb4e-198e97a2c5ff"). InnerVolumeSpecName "kube-api-access-njb7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:28.358295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.358260    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca" pod "d025f59e-4a71-4801-bb4e-198e97a2c5ff" (UID: "d025f59e-4a71-4801-bb4e-198e97a2c5ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:48:28.465497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.465340    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025f59e-4a71-4801-bb4e-198e97a2c5ff-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:28.465497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:28.465380    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njb7r\" (UniqueName: \"kubernetes.io/projected/d025f59e-4a71-4801-bb4e-198e97a2c5ff-kube-api-access-njb7r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:29.045548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.045447    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk" event={"ID":"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d","Type":"ContainerStarted","Data":"346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238"}
Oct 30 17:48:29.045763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.045549    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk" event={"ID":"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d","Type":"ContainerStarted","Data":"0d86d00127260c44350b5a4ee00dbcdb4cb028863652af3f2c0c737e891dd002"}
Oct 30 17:48:29.045763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.045708    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:29.047420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.047395    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"
Oct 30 17:48:29.047677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.047415    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6" event={"ID":"d025f59e-4a71-4801-bb4e-198e97a2c5ff","Type":"ContainerDied","Data":"7721fa353fa9b3e5801f96a4fc86c0499f82d3905af789104d1193fb6e69be4d"}
Oct 30 17:48:29.047677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.047465    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7721fa353fa9b3e5801f96a4fc86c0499f82d3905af789104d1193fb6e69be4d"
Oct 30 17:48:29.324225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.324026    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk" podStartSLOduration=1.934679802 podStartE2EDuration="2.324003294s" totalImagesPullingTime="389.323492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:29.107005408 +0000 UTC m=+2114.869049225" watchObservedRunningTime="2026-10-30 17:48:29.324003294 +0000 UTC m=+2115.086047108"
Oct 30 17:48:29.667713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.657942    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:29.923275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:29.921567    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:30.082522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.082483    5534 generic.go:356] "Generic (PLEG): container finished" podID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerID="a05a16fde58f72fbf16e5e37d8240d09f176bbc1b021ca54fd85da89c44d30dc" exitCode=0
Oct 30 17:48:30.083820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.083792    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" event={"ID":"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439","Type":"ContainerDied","Data":"a05a16fde58f72fbf16e5e37d8240d09f176bbc1b021ca54fd85da89c44d30dc"}
Oct 30 17:48:30.084016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.084001    5534 scope.go:122] "RemoveContainer" containerID="f1baf28d949b9804f2b36ef43e248c529f846650008ca9d53f9037295b3cc6ae"
Oct 30 17:48:30.441703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.441679    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:48:30.539520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539428    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.539713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539541    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.539713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539579    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.539852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539818    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.539917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539854    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.539917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539815    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:48:30.539917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539907    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.540077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539906    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:48:30.540077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.539985    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.540077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540048    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.540225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540074    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms\") pod \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\" (UID: \"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439\") "
Oct 30 17:48:30.540225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540178    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:48:30.540326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540258    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:48:30.540392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540375    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:48:30.540786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540763    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.540786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540787    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.540998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540801    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.540998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540813    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.540998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.540826    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.543728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.543256    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:48:30.543728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.543684    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:48:30.544471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.544438    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "kube-api-access-pp8ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:30.544587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.544479    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token" pod "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" (UID: "1b94e65d-aff5-45f9-a4eb-8a54a4eaf439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:30.642972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.642942    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.642972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.642969    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.642972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.642980    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8ms\" (UniqueName: \"kubernetes.io/projected/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-kube-api-access-pp8ms\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:30.643231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:30.642988    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:31.094555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:31.094526    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"
Oct 30 17:48:31.094555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:31.094565    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj" event={"ID":"1b94e65d-aff5-45f9-a4eb-8a54a4eaf439","Type":"ContainerDied","Data":"c8242c79ccc06da71a522e6a31c4520b17aff87dd6d811c53dda7b2e710e15da"}
Oct 30 17:48:31.094871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:31.094678    5534 scope.go:122] "RemoveContainer" containerID="a05a16fde58f72fbf16e5e37d8240d09f176bbc1b021ca54fd85da89c44d30dc"
Oct 30 17:48:31.167441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:31.167340    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"]
Oct 30 17:48:31.176871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:31.176844    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-5kmnj"]
Oct 30 17:48:31.528609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:31.528523    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:32.377906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:32.377869    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" path="/var/lib/kubelet/pods/1b94e65d-aff5-45f9-a4eb-8a54a4eaf439/volumes"
Oct 30 17:48:33.387787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:33.387745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:35.054249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:35.054215    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:35.090691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:35.090624    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:48:36.485742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:36.485703    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"]
Oct 30 17:48:36.600367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:36.600296    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:36.710862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:36.710828    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:38.592940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:38.592911    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:40.251977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:40.251922    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:40.377761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:40.374753    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:41.491066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:41.491022    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerName="payload-processing" containerID="cri-o://56e8d14e1c84c021e346abd2eb00fc10f7708e5d2d25fe3c84fa5c606110c4af" gracePeriod=25
Oct 30 17:48:42.050609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:42.050576    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:43.961153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:43.961119    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:44.216150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:44.216066    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:48:45.247293 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:48:45.247257    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:48:45.570741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:45.570608    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:47.236715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:47.236682    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:48.271597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:48.271564    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:48.385910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:48.385879    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:49.016795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:49.016764    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:50.694788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:50.694743    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:52.347331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:52.347294    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:53.683363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.683331    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"]
Oct 30 17:48:53.684398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684367    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d025f59e-4a71-4801-bb4e-198e97a2c5ff" containerName="revoke-keys"
Oct 30 17:48:53.684398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684389    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d025f59e-4a71-4801-bb4e-198e97a2c5ff" containerName="revoke-keys"
Oct 30 17:48:53.684398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684404    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy"
Oct 30 17:48:53.684578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684410    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy"
Oct 30 17:48:53.684620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684587    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy"
Oct 30 17:48:53.684620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684603    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d025f59e-4a71-4801-bb4e-198e97a2c5ff" containerName="revoke-keys"
Oct 30 17:48:53.684620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.684611    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy"
Oct 30 17:48:53.689571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.689552    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.707142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.707113    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"]
Oct 30 17:48:53.712799 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.712771    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:48:53.712977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.712860    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:48:53.803684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.803623    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgqr\" (UniqueName: \"kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.804037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.804018    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.804353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.804330    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.906538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.906502    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgqr\" (UniqueName: \"kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.906736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.906583    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.906804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.906766    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.907904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.907877    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.909528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.909503    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:53.929754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:53.929731    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgqr\" (UniqueName: \"kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr\") pod \"maas-api-revoke-sub-fc38ecd5-z7qtv\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:54.006351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.006262    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:54.216150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.216111    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:48:54.296421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.296334    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:54.317206 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:48:54.317177    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f1ba1c_1014_4c63_bbe4_eabb980fc254.slice/crio-8de4800b9e9f69eef49eabe563a4ab6272b7624f199eef381ff365e460af1db2 WatchSource:0}: Error finding container 8de4800b9e9f69eef49eabe563a4ab6272b7624f199eef381ff365e460af1db2: Status 404 returned error can't find the container with id 8de4800b9e9f69eef49eabe563a4ab6272b7624f199eef381ff365e460af1db2
Oct 30 17:48:54.317319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.317216    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"]
Oct 30 17:48:54.381305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.381263    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv" event={"ID":"83f1ba1c-1014-4c63-bbe4-eabb980fc254","Type":"ContainerStarted","Data":"8de4800b9e9f69eef49eabe563a4ab6272b7624f199eef381ff365e460af1db2"}
Oct 30 17:48:54.539631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.539601    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"]
Oct 30 17:48:54.540452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.540437    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy"
Oct 30 17:48:54.540506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.540454    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b94e65d-aff5-45f9-a4eb-8a54a4eaf439" containerName="istio-proxy"
Oct 30 17:48:54.545865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.545848    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.564599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.564574    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"]
Oct 30 17:48:54.619889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.619859    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qz8q\" (UniqueName: \"kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.620187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.620166    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.620307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.620238    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.722120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.722079    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.722622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.722156    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.722622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.722213    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qz8q\" (UniqueName: \"kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.722931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.722911    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.724739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.724715    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.745470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.745439    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qz8q\" (UniqueName: \"kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q\") pod \"maas-api-revoke-sub-08e46ca3-lftrt\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:54.863782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:54.863751    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:55.182546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:55.182470    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"]
Oct 30 17:48:55.187462 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:48:55.187432    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff00a7e_4ba0_4110_8d54_fcb543ec6495.slice/crio-64f59682962c4dd83a7dcaeaf3947d2c6e9671b5f64e412e13eb50fd475c164e WatchSource:0}: Error finding container 64f59682962c4dd83a7dcaeaf3947d2c6e9671b5f64e412e13eb50fd475c164e: Status 404 returned error can't find the container with id 64f59682962c4dd83a7dcaeaf3947d2c6e9671b5f64e412e13eb50fd475c164e
Oct 30 17:48:55.390415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:55.390378    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv" event={"ID":"83f1ba1c-1014-4c63-bbe4-eabb980fc254","Type":"ContainerStarted","Data":"dfffa67e179644e7e7d0b0b3fa15e15fc50853f465d7f504781ebacb3406fc2b"}
Oct 30 17:48:55.391987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:55.391959    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt" event={"ID":"fff00a7e-4ba0-4110-8d54-fcb543ec6495","Type":"ContainerStarted","Data":"64f59682962c4dd83a7dcaeaf3947d2c6e9671b5f64e412e13eb50fd475c164e"}
Oct 30 17:48:55.431332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:55.431284    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv" podStartSLOduration=2.431270962 podStartE2EDuration="2.431270962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:55.429086431 +0000 UTC m=+2141.191130245" watchObservedRunningTime="2026-10-30 17:48:55.431270962 +0000 UTC m=+2141.193314774"
Oct 30 17:48:55.970566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:55.970528    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:56.656951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:56.656915    5534 generic.go:356] "Generic (PLEG): container finished" podID="83f1ba1c-1014-4c63-bbe4-eabb980fc254" containerID="dfffa67e179644e7e7d0b0b3fa15e15fc50853f465d7f504781ebacb3406fc2b" exitCode=0
Oct 30 17:48:56.657429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:56.657400    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv" event={"ID":"83f1ba1c-1014-4c63-bbe4-eabb980fc254","Type":"ContainerDied","Data":"dfffa67e179644e7e7d0b0b3fa15e15fc50853f465d7f504781ebacb3406fc2b"}
Oct 30 17:48:56.665014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:56.664899    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt" event={"ID":"fff00a7e-4ba0-4110-8d54-fcb543ec6495","Type":"ContainerStarted","Data":"49e9eb997433c7b37990f645808952fc4d30f4a4bcb572299eae5d58a4fc0195"}
Oct 30 17:48:57.676786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:57.676744    5534 generic.go:356] "Generic (PLEG): container finished" podID="fff00a7e-4ba0-4110-8d54-fcb543ec6495" containerID="49e9eb997433c7b37990f645808952fc4d30f4a4bcb572299eae5d58a4fc0195" exitCode=0
Oct 30 17:48:57.677291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:57.677160    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt" event={"ID":"fff00a7e-4ba0-4110-8d54-fcb543ec6495","Type":"ContainerDied","Data":"49e9eb997433c7b37990f645808952fc4d30f4a4bcb572299eae5d58a4fc0195"}
Oct 30 17:48:57.878713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:57.872230    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:48:58.178892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.178866    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:58.237940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.237249    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca\") pod \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") "
Oct 30 17:48:58.237940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.237353    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca\") pod \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") "
Oct 30 17:48:58.237940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.237608    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr\" (UniqueName: \"kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr\") pod \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\" (UID: \"83f1ba1c-1014-4c63-bbe4-eabb980fc254\") "
Oct 30 17:48:58.240117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.238984    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca" pod "83f1ba1c-1014-4c63-bbe4-eabb980fc254" (UID: "83f1ba1c-1014-4c63-bbe4-eabb980fc254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:48:58.242607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.241227    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr" pod "83f1ba1c-1014-4c63-bbe4-eabb980fc254" (UID: "83f1ba1c-1014-4c63-bbe4-eabb980fc254"). InnerVolumeSpecName "kube-api-access-qmgqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:58.243566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.243502    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca" pod "83f1ba1c-1014-4c63-bbe4-eabb980fc254" (UID: "83f1ba1c-1014-4c63-bbe4-eabb980fc254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:48:58.281407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.277406    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:58.339497 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.339462    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca\") pod \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") "
Oct 30 17:48:58.339764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.339521    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca\") pod \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") "
Oct 30 17:48:58.339764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.339725    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q\" (UniqueName: \"kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q\") pod \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\" (UID: \"fff00a7e-4ba0-4110-8d54-fcb543ec6495\") "
Oct 30 17:48:58.340381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.340354    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmgqr\" (UniqueName: \"kubernetes.io/projected/83f1ba1c-1014-4c63-bbe4-eabb980fc254-kube-api-access-qmgqr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:58.340505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.340386    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:58.340505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.340403    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f1ba1c-1014-4c63-bbe4-eabb980fc254-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:58.341151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.340925    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca" pod "fff00a7e-4ba0-4110-8d54-fcb543ec6495" (UID: "fff00a7e-4ba0-4110-8d54-fcb543ec6495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:48:58.343125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.343092    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q" pod "fff00a7e-4ba0-4110-8d54-fcb543ec6495" (UID: "fff00a7e-4ba0-4110-8d54-fcb543ec6495"). InnerVolumeSpecName "kube-api-access-2qz8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:48:58.343469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.343447    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca" pod "fff00a7e-4ba0-4110-8d54-fcb543ec6495" (UID: "fff00a7e-4ba0-4110-8d54-fcb543ec6495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:48:58.441988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.441953    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:58.441988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.441991    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff00a7e-4ba0-4110-8d54-fcb543ec6495-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:58.442243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.442014    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qz8q\" (UniqueName: \"kubernetes.io/projected/fff00a7e-4ba0-4110-8d54-fcb543ec6495-kube-api-access-2qz8q\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:48:58.759836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.759794    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv" event={"ID":"83f1ba1c-1014-4c63-bbe4-eabb980fc254","Type":"ContainerDied","Data":"8de4800b9e9f69eef49eabe563a4ab6272b7624f199eef381ff365e460af1db2"}
Oct 30 17:48:58.760477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.760456    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de4800b9e9f69eef49eabe563a4ab6272b7624f199eef381ff365e460af1db2"
Oct 30 17:48:58.760701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.760686    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"
Oct 30 17:48:58.783349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.783313    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt" event={"ID":"fff00a7e-4ba0-4110-8d54-fcb543ec6495","Type":"ContainerDied","Data":"64f59682962c4dd83a7dcaeaf3947d2c6e9671b5f64e412e13eb50fd475c164e"}
Oct 30 17:48:58.783802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.783785    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f59682962c4dd83a7dcaeaf3947d2c6e9671b5f64e412e13eb50fd475c164e"
Oct 30 17:48:58.783941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:58.783613    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"
Oct 30 17:48:59.826607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:48:59.826577    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:00.965787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:00.965755    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:01.540685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.535567    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:01.809141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.809107    5534 generic.go:356] "Generic (PLEG): container finished" podID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerID="56e8d14e1c84c021e346abd2eb00fc10f7708e5d2d25fe3c84fa5c606110c4af" exitCode=0
Oct 30 17:49:01.809319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.809188    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" event={"ID":"aec1270c-2741-49f4-9e21-9d53138ad2d7","Type":"ContainerDied","Data":"56e8d14e1c84c021e346abd2eb00fc10f7708e5d2d25fe3c84fa5c606110c4af"}
Oct 30 17:49:01.809319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.809244    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt" event={"ID":"aec1270c-2741-49f4-9e21-9d53138ad2d7","Type":"ContainerDied","Data":"b03c8adfd11ad372ce9d92650218267beb400a2c01c9d6c52b869e1dd20ba2c2"}
Oct 30 17:49:01.809319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.809269    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03c8adfd11ad372ce9d92650218267beb400a2c01c9d6c52b869e1dd20ba2c2"
Oct 30 17:49:01.839969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.839797    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:49:01.943889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.943860    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g\" (UniqueName: \"kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g\") pod \"aec1270c-2741-49f4-9e21-9d53138ad2d7\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") "
Oct 30 17:49:01.944201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.944182    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume\") pod \"aec1270c-2741-49f4-9e21-9d53138ad2d7\" (UID: \"aec1270c-2741-49f4-9e21-9d53138ad2d7\") "
Oct 30 17:49:01.944578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.944555    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume" pod "aec1270c-2741-49f4-9e21-9d53138ad2d7" (UID: "aec1270c-2741-49f4-9e21-9d53138ad2d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:01.944858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.944818    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec1270c-2741-49f4-9e21-9d53138ad2d7-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:01.946689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:01.946666    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g" pod "aec1270c-2741-49f4-9e21-9d53138ad2d7" (UID: "aec1270c-2741-49f4-9e21-9d53138ad2d7"). InnerVolumeSpecName "kube-api-access-k6d5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:02.045985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:02.045950    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6d5g\" (UniqueName: \"kubernetes.io/projected/aec1270c-2741-49f4-9e21-9d53138ad2d7-kube-api-access-k6d5g\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:02.819490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:02.819388    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"
Oct 30 17:49:02.872662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:02.872613    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"]
Oct 30 17:49:02.884945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:02.884914    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-143041-5bbd8bd9d8-7jdmt"]
Oct 30 17:49:03.311662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:03.311620    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:04.373741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:04.373709    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" path="/var/lib/kubelet/pods/aec1270c-2741-49f4-9e21-9d53138ad2d7/volumes"
Oct 30 17:49:04.654531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:04.654443    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:05.018813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.018729    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:05.215252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.215215    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"]
Oct 30 17:49:05.216392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216375    5534 cpu_manager.go:414] "containerMap: removing container" podUID="83f1ba1c-1014-4c63-bbe4-eabb980fc254" containerName="revoke-keys"
Oct 30 17:49:05.216447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216395    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f1ba1c-1014-4c63-bbe4-eabb980fc254" containerName="revoke-keys"
Oct 30 17:49:05.216447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216428    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fff00a7e-4ba0-4110-8d54-fcb543ec6495" containerName="revoke-keys"
Oct 30 17:49:05.216447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216437    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff00a7e-4ba0-4110-8d54-fcb543ec6495" containerName="revoke-keys"
Oct 30 17:49:05.216553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216450    5534 cpu_manager.go:414] "containerMap: removing container" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerName="payload-processing"
Oct 30 17:49:05.216553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216458    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerName="payload-processing"
Oct 30 17:49:05.216636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216625    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f1ba1c-1014-4c63-bbe4-eabb980fc254" containerName="revoke-keys"
Oct 30 17:49:05.216636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216661    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff00a7e-4ba0-4110-8d54-fcb543ec6495" containerName="revoke-keys"
Oct 30 17:49:05.216741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.216671    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec1270c-2741-49f4-9e21-9d53138ad2d7" containerName="payload-processing"
Oct 30 17:49:05.220921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.220905    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.238194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.238172    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"]
Oct 30 17:49:05.245031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.244996    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:49:05.245031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.244996    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:49:05.263982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.263950    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.264153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.264099    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trx8\" (UniqueName: \"kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.264255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.264237    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.365735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.365690    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.365925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.365761    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trx8\" (UniqueName: \"kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.365925 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.365819    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.366600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.366575    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.368291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.368270    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.389470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.389440    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trx8\" (UniqueName: \"kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8\") pod \"maas-api-revoke-sub-7047efaf-b8lsn\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.535533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.535503    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:05.848785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.848755    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"]
Oct 30 17:49:05.850147 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:49:05.850123    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad01cb1_91b5_4975_aa1f_52e582e75e06.slice/crio-4bef41764fe77c2983fb244cce40d8e8961413ea713c77fc8acb153b4b22132b WatchSource:0}: Error finding container 4bef41764fe77c2983fb244cce40d8e8961413ea713c77fc8acb153b4b22132b: Status 404 returned error can't find the container with id 4bef41764fe77c2983fb244cce40d8e8961413ea713c77fc8acb153b4b22132b
Oct 30 17:49:05.944872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:05.944845    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:06.631837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:06.631803    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:06.746843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:06.746805    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:06.863975 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:06.863934    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn" event={"ID":"2ad01cb1-91b5-4975-aa1f-52e582e75e06","Type":"ContainerStarted","Data":"d9fde37ae928ea9e3abd69c0c608dfd9df1af5b0193f02e14a529f3d102d7ea4"}
Oct 30 17:49:06.864163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:06.864032    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn" event={"ID":"2ad01cb1-91b5-4975-aa1f-52e582e75e06","Type":"ContainerStarted","Data":"4bef41764fe77c2983fb244cce40d8e8961413ea713c77fc8acb153b4b22132b"}
Oct 30 17:49:06.905521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:06.905474    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn" podStartSLOduration=1.90546074 podStartE2EDuration="1.90546074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:06.903612495 +0000 UTC m=+2152.665656310" watchObservedRunningTime="2026-10-30 17:49:06.90546074 +0000 UTC m=+2152.667504554"
Oct 30 17:49:07.282060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:07.281982    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:07.874274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:07.874238    5534 generic.go:356] "Generic (PLEG): container finished" podID="2ad01cb1-91b5-4975-aa1f-52e582e75e06" containerID="d9fde37ae928ea9e3abd69c0c608dfd9df1af5b0193f02e14a529f3d102d7ea4" exitCode=0
Oct 30 17:49:07.874785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:07.874285    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn" event={"ID":"2ad01cb1-91b5-4975-aa1f-52e582e75e06","Type":"ContainerDied","Data":"d9fde37ae928ea9e3abd69c0c608dfd9df1af5b0193f02e14a529f3d102d7ea4"}
Oct 30 17:49:08.089196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:08.089166    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:08.198511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:08.198424    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:08.695660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:08.691468    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:08.915830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:08.915798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:08.991435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:08.991367    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"]
Oct 30 17:49:08.997982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:08.997951    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-143041-b9sh6"]
Oct 30 17:49:09.033686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.027228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:09.150027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.150003    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:09.247716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.247610    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8\" (UniqueName: \"kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8\") pod \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") "
Oct 30 17:49:09.247899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.247724    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca\") pod \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") "
Oct 30 17:49:09.247899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.247798    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca\") pod \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\" (UID: \"2ad01cb1-91b5-4975-aa1f-52e582e75e06\") "
Oct 30 17:49:09.248433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.248371    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca" pod "2ad01cb1-91b5-4975-aa1f-52e582e75e06" (UID: "2ad01cb1-91b5-4975-aa1f-52e582e75e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:09.248747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.248722    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:09.250699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.250618    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8" pod "2ad01cb1-91b5-4975-aa1f-52e582e75e06" (UID: "2ad01cb1-91b5-4975-aa1f-52e582e75e06"). InnerVolumeSpecName "kube-api-access-4trx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:09.250802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.250760    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca" pod "2ad01cb1-91b5-4975-aa1f-52e582e75e06" (UID: "2ad01cb1-91b5-4975-aa1f-52e582e75e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:49:09.350411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.350376    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trx8\" (UniqueName: \"kubernetes.io/projected/2ad01cb1-91b5-4975-aa1f-52e582e75e06-kube-api-access-4trx8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:09.350411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.350406    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad01cb1-91b5-4975-aa1f-52e582e75e06-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:09.491862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.491816    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:09.894819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.894790    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"
Oct 30 17:49:09.895023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.894832    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn" event={"ID":"2ad01cb1-91b5-4975-aa1f-52e582e75e06","Type":"ContainerDied","Data":"4bef41764fe77c2983fb244cce40d8e8961413ea713c77fc8acb153b4b22132b"}
Oct 30 17:49:09.895023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:09.894883    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bef41764fe77c2983fb244cce40d8e8961413ea713c77fc8acb153b4b22132b"
Oct 30 17:49:10.321964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:10.321857    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:10.374074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:10.374044    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d025f59e-4a71-4801-bb4e-198e97a2c5ff" path="/var/lib/kubelet/pods/d025f59e-4a71-4801-bb4e-198e97a2c5ff/volumes"
Oct 30 17:49:10.473159 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:10.473125    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:10.790391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:10.790357    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:11.372686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:11.372654    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"]
Oct 30 17:49:11.373099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:11.372824    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" containerID="cri-o://ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2" gracePeriod=30
Oct 30 17:49:12.614042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:12.614013    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:14.333289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:14.333262    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:15.245773 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:15.245734    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:49:15.843675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:15.842396    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:16.006915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.002886    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:16.055534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.055496    5534 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:49:16.056919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.055819    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:49:16.175090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.174893    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"]
Oct 30 17:49:16.175689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.175633    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy" containerID="cri-o://f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d" gracePeriod=30
Oct 30 17:49:16.222450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.222414    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:16.354666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.354270    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:16.793381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.793350    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:49:16.919440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919408    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.919913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919476    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.919913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919501    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.919913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919523    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.919913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919628    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.919913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919697    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.919913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919748    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.920258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.919995    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:16.920258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.920084    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.920258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.920093    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:16.920530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.920512    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:16.920608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.920558    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:16.920947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.920920    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:16.921004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.920967    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data\") pod \"9cd52897-715d-4d7b-b3ef-79aac18b7226\" (UID: \"9cd52897-715d-4d7b-b3ef-79aac18b7226\") "
Oct 30 17:49:16.921887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.921867    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:16.921941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.921897    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:16.921941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.921914    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd52897-715d-4d7b-b3ef-79aac18b7226-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:16.921941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.921927    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:16.922046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.921942    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:16.923045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.923019    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "kube-api-access-vdmrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:16.923231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.923207    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:16.924298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.924223    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:16.924778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:16.924751    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo" pod "9cd52897-715d-4d7b-b3ef-79aac18b7226" (UID: "9cd52897-715d-4d7b-b3ef-79aac18b7226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:49:17.001706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.001665    5534 generic.go:356] "Generic (PLEG): container finished" podID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerID="ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2" exitCode=0
Oct 30 17:49:17.001894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.001741    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"
Oct 30 17:49:17.001979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.001960    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" event={"ID":"9cd52897-715d-4d7b-b3ef-79aac18b7226","Type":"ContainerDied","Data":"ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2"}
Oct 30 17:49:17.002082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.002017    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn" event={"ID":"9cd52897-715d-4d7b-b3ef-79aac18b7226","Type":"ContainerDied","Data":"97115eb1b65ea11b173077a1b0bed6d012f72ab05ef1db8dc64ec01510ddd3b3"}
Oct 30 17:49:17.002082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.002049    5534 scope.go:122] "RemoveContainer" containerID="ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2"
Oct 30 17:49:17.018872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.018828    5534 scope.go:122] "RemoveContainer" containerID="ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2"
Oct 30 17:49:17.019485 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:17.019383    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2\": container with ID starting with ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2 not found: ID does not exist" containerID="ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2"
Oct 30 17:49:17.019485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.019422    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2"} err="failed to get container status \"ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2\": rpc error: code = NotFound desc = could not find container \"ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2\": container with ID starting with ea4e637747d5f9bab64467605cad19e786c54c59c8fe0d1c94bd255264c64da2 not found: ID does not exist"
Oct 30 17:49:17.033820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.024576    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:17.033820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.024603    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdmrl\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-kube-api-access-vdmrl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:17.033820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.024619    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:17.033820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.024636    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd52897-715d-4d7b-b3ef-79aac18b7226-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:17.059930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.059897    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"]
Oct 30 17:49:17.078009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.077978    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-mqvgn"]
Oct 30 17:49:17.102359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.102319    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:17.313682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.313579    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:17.811319 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.811289    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:17.929626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.928980    5534 scope.go:122] "RemoveContainer" containerID="f8aedcdba41bd9cfaf6c6eed33bcd884bf07865a1bf3c0a32db986eb36d06583"
Oct 30 17:49:17.947678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.947624    5534 scope.go:122] "RemoveContainer" containerID="622878a54d6d239213873ce6aff96aaee6af8c8599459c06fbc50709821c3acf"
Oct 30 17:49:17.995954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:17.995917    5534 scope.go:122] "RemoveContainer" containerID="56e8d14e1c84c021e346abd2eb00fc10f7708e5d2d25fe3c84fa5c606110c4af"
Oct 30 17:49:18.224687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:18.224653    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:18.334197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:18.334167    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:18.379176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:18.379138    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" path="/var/lib/kubelet/pods/9cd52897-715d-4d7b-b3ef-79aac18b7226/volumes"
Oct 30 17:49:19.576923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:19.576884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:20.112034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:20.112005    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:21.530706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.530681    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:49:21.626146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626109    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.626362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626258    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.626362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626300    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.626362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626345    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.626546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626417    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:21.626807 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626776    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.626928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.626851    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.627148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.627125    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:21.627422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.627400    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.627547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.627525    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.628337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.627611    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket\") pod \"74ccb873-6839-424e-b006-d07d6bc224a9\" (UID: \"74ccb873-6839-424e-b006-d07d6bc224a9\") "
Oct 30 17:49:21.628337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.628016    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:21.628337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.628287    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:21.628717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.628693    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.628809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.628722    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccb873-6839-424e-b006-d07d6bc224a9-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.628809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.628736    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.628809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.628749    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.629092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.629070    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:21.631675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.629869    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:21.631675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.630013    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "kube-api-access-jxdhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:21.632819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.632794    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:21.634827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.634792    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo" pod "74ccb873-6839-424e-b006-d07d6bc224a9" (UID: "74ccb873-6839-424e-b006-d07d6bc224a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:49:21.730107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.730064    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccb873-6839-424e-b006-d07d6bc224a9-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.730107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.730103    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdhh\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-kube-api-access-jxdhh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.730107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.730113    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.730371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.730123    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccb873-6839-424e-b006-d07d6bc224a9-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.730371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.730132    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccb873-6839-424e-b006-d07d6bc224a9-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:21.829686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.829591    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:21.982569 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:21.982494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:22.066139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.066107    5534 generic.go:356] "Generic (PLEG): container finished" podID="74ccb873-6839-424e-b006-d07d6bc224a9" containerID="f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d" exitCode=0
Oct 30 17:49:22.066362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.066160    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"
Oct 30 17:49:22.066362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.066185    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" event={"ID":"74ccb873-6839-424e-b006-d07d6bc224a9","Type":"ContainerDied","Data":"f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d"}
Oct 30 17:49:22.066362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.066252    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z" event={"ID":"74ccb873-6839-424e-b006-d07d6bc224a9","Type":"ContainerDied","Data":"0dbbef06d787d6c7c5fa402af5367d0c245efcd8f19683b445a27bf6e20ba308"}
Oct 30 17:49:22.066362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.066281    5534 scope.go:122] "RemoveContainer" containerID="f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d"
Oct 30 17:49:22.080433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.080403    5534 scope.go:122] "RemoveContainer" containerID="1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"
Oct 30 17:49:22.091741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.091714    5534 scope.go:122] "RemoveContainer" containerID="f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d"
Oct 30 17:49:22.092101 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:22.092076    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d\": container with ID starting with f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d not found: ID does not exist" containerID="f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d"
Oct 30 17:49:22.092184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.092109    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d"} err="failed to get container status \"f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d\": rpc error: code = NotFound desc = could not find container \"f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d\": container with ID starting with f0afb349f6796911d4db77b386edc13e1d98937bef95d3cec6da01eaedb1cf0d not found: ID does not exist"
Oct 30 17:49:22.092184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.092130    5534 scope.go:122] "RemoveContainer" containerID="1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"
Oct 30 17:49:22.092443 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:22.092427    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f\": container with ID starting with 1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f not found: ID does not exist" containerID="1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"
Oct 30 17:49:22.092488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.092452    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f"} err="failed to get container status \"1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f\": rpc error: code = NotFound desc = could not find container \"1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f\": container with ID starting with 1dc91c6f88a88254d79b941c070dde418248822aadca179eb80b95c20d08866f not found: ID does not exist"
Oct 30 17:49:22.124729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.124692    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"]
Oct 30 17:49:22.138698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.138669    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-8446z"]
Oct 30 17:49:22.142755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.142730    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:22.266856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.266761    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:22.372621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:22.372587    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" path="/var/lib/kubelet/pods/74ccb873-6839-424e-b006-d07d6bc224a9/volumes"
Oct 30 17:49:23.531738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.531702    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:23.644525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.644494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:23.697467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.697434    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"]
Oct 30 17:49:23.698298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698276    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2ad01cb1-91b5-4975-aa1f-52e582e75e06" containerName="revoke-keys"
Oct 30 17:49:23.698298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698297    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad01cb1-91b5-4975-aa1f-52e582e75e06" containerName="revoke-keys"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698324    5534 cpu_manager.go:414] "containerMap: removing container" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698333    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698350    5534 cpu_manager.go:414] "containerMap: removing container" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698355    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698370    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698375    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy"
Oct 30 17:49:23.698517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698521    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad01cb1-91b5-4975-aa1f-52e582e75e06" containerName="revoke-keys"
Oct 30 17:49:23.698929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698533    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy"
Oct 30 17:49:23.698929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698541    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ccb873-6839-424e-b006-d07d6bc224a9" containerName="istio-proxy"
Oct 30 17:49:23.698929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.698548    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd52897-715d-4d7b-b3ef-79aac18b7226" containerName="istio-proxy"
Oct 30 17:49:23.703677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.703635    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:23.722256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.722225    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"]
Oct 30 17:49:23.735055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.735017    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ce0480-dockercfg-srldd\""
Oct 30 17:49:23.735919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.735601    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ce0480\""
Oct 30 17:49:23.829716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.828548    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjzj\" (UniqueName: \"kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj\") pod \"payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:23.829716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:23.828914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:24.009666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.008634    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:24.009666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.008834    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjzj\" (UniqueName: \"kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj\") pod \"payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:24.025667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.010511    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:24.056008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.053469    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjzj\" (UniqueName: \"kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj\") pod \"payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:24.303277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.303218    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:24.321859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.321817    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:24.688715 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:49:24.685798    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf065fa93_9947_4eba_8440_5ead8e5c2188.slice/crio-adbcc4a63dac5408be6675ea72db95f037ab333d1870378bd79556bdc7e956dc WatchSource:0}: Error finding container adbcc4a63dac5408be6675ea72db95f037ab333d1870378bd79556bdc7e956dc: Status 404 returned error can't find the container with id adbcc4a63dac5408be6675ea72db95f037ab333d1870378bd79556bdc7e956dc
Oct 30 17:49:24.689280 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:24.688794    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"]
Oct 30 17:49:25.107940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:25.107901    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" event={"ID":"f065fa93-9947-4eba-8440-5ead8e5c2188","Type":"ContainerStarted","Data":"dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b"}
Oct 30 17:49:25.108170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:25.107959    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" event={"ID":"f065fa93-9947-4eba-8440-5ead8e5c2188","Type":"ContainerStarted","Data":"adbcc4a63dac5408be6675ea72db95f037ab333d1870378bd79556bdc7e956dc"}
Oct 30 17:49:25.108170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:25.108138    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:25.151275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:25.151223    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" podStartSLOduration=2.151209825 podStartE2EDuration="2.151209825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:25.147721201 +0000 UTC m=+2170.909765015" watchObservedRunningTime="2026-10-30 17:49:25.151209825 +0000 UTC m=+2170.913253674"
Oct 30 17:49:26.102942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:26.102894    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:27.875079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:27.874965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:28.502613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:28.502568    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:29.038276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.038242    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"]
Oct 30 17:49:29.044155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.044137    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.064369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.064342    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"]
Oct 30 17:49:29.070478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.070454    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:49:29.070689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.070595    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:49:29.110927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.110887    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9487z\" (UniqueName: \"kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.111115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.111024    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.111278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.111259    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.213250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.213213    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.213461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.213315    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9487z\" (UniqueName: \"kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.213461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.213363    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.214037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.214014    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.215946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.215925    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.236787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.236762    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9487z\" (UniqueName: \"kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z\") pod \"maas-api-revoke-sub-d4e9ad10-wdwv2\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.360991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.360952    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:29.668366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.668286    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:29.931598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:29.931566    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"]
Oct 30 17:49:30.161821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:30.161784    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2" event={"ID":"89ff5baf-45f4-4742-9d01-0c290eb06231","Type":"ContainerStarted","Data":"af86c53d7851b9d01c357f90cb1068e6ffbc7c20fb4bbdec27dc1af0f4069d9d"}
Oct 30 17:49:31.176101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:31.176053    5534 generic.go:356] "Generic (PLEG): container finished" podID="89ff5baf-45f4-4742-9d01-0c290eb06231" containerID="9484688bbb0b136d4aeb36ae198759f9e9e5cef66f8ecb48a34d6cd5014ab730" exitCode=0
Oct 30 17:49:31.176660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:31.176394    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2" event={"ID":"89ff5baf-45f4-4742-9d01-0c290eb06231","Type":"ContainerDied","Data":"9484688bbb0b136d4aeb36ae198759f9e9e5cef66f8ecb48a34d6cd5014ab730"}
Oct 30 17:49:31.595939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:31.595904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:32.709428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.709398    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:32.722311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.722283    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z\" (UniqueName: \"kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z\") pod \"89ff5baf-45f4-4742-9d01-0c290eb06231\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") "
Oct 30 17:49:32.722485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.722359    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca\") pod \"89ff5baf-45f4-4742-9d01-0c290eb06231\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") "
Oct 30 17:49:32.722485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.722380    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca\") pod \"89ff5baf-45f4-4742-9d01-0c290eb06231\" (UID: \"89ff5baf-45f4-4742-9d01-0c290eb06231\") "
Oct 30 17:49:32.723272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.723233    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca" pod "89ff5baf-45f4-4742-9d01-0c290eb06231" (UID: "89ff5baf-45f4-4742-9d01-0c290eb06231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:32.728032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.728007    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca" pod "89ff5baf-45f4-4742-9d01-0c290eb06231" (UID: "89ff5baf-45f4-4742-9d01-0c290eb06231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:49:32.728334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.728303    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z" pod "89ff5baf-45f4-4742-9d01-0c290eb06231" (UID: "89ff5baf-45f4-4742-9d01-0c290eb06231"). InnerVolumeSpecName "kube-api-access-9487z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:32.825492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.825453    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9487z\" (UniqueName: \"kubernetes.io/projected/89ff5baf-45f4-4742-9d01-0c290eb06231-kube-api-access-9487z\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:32.825492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.825495    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:32.825742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:32.825512    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ff5baf-45f4-4742-9d01-0c290eb06231-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:33.228225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:33.228189    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2" event={"ID":"89ff5baf-45f4-4742-9d01-0c290eb06231","Type":"ContainerDied","Data":"af86c53d7851b9d01c357f90cb1068e6ffbc7c20fb4bbdec27dc1af0f4069d9d"}
Oct 30 17:49:33.228225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:33.228215    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"
Oct 30 17:49:33.228454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:33.228253    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af86c53d7851b9d01c357f90cb1068e6ffbc7c20fb4bbdec27dc1af0f4069d9d"
Oct 30 17:49:33.499218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:33.499138    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:34.184076 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.184037    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:34.399237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.399207    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"]
Oct 30 17:49:34.400042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.400028    5534 cpu_manager.go:414] "containerMap: removing container" podUID="89ff5baf-45f4-4742-9d01-0c290eb06231" containerName="revoke-keys"
Oct 30 17:49:34.400116 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.400043    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ff5baf-45f4-4742-9d01-0c290eb06231" containerName="revoke-keys"
Oct 30 17:49:34.400210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.400159    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ff5baf-45f4-4742-9d01-0c290eb06231" containerName="revoke-keys"
Oct 30 17:49:34.404219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.404198    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.432704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.432675    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:49:34.450186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.450123    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"]
Oct 30 17:49:34.462605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.462576    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.467788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.464747    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.467788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.464860    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9vf\" (UniqueName: \"kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.535038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.535005    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:49:34.566782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.566744    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.566983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.566791    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.566983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.566851    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9vf\" (UniqueName: \"kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.567856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.567830    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.570008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.569982    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.591163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.591127    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9vf\" (UniqueName: \"kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf\") pod \"maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:34.719085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:34.718993    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:35.036416 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:49:35.036387    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ca6b6_54bb_459a_a90d_2648bf6001bd.slice/crio-fa6ac96c618ee0efd35f9d9aa1a5e6bc22e379d2ff0afa2fc693887bdb607238 WatchSource:0}: Error finding container fa6ac96c618ee0efd35f9d9aa1a5e6bc22e379d2ff0afa2fc693887bdb607238: Status 404 returned error can't find the container with id fa6ac96c618ee0efd35f9d9aa1a5e6bc22e379d2ff0afa2fc693887bdb607238
Oct 30 17:49:35.036572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:35.036542    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"]
Oct 30 17:49:35.255594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:35.255558    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4" event={"ID":"257ca6b6-54bb-459a-a90d-2648bf6001bd","Type":"ContainerStarted","Data":"fa6ac96c618ee0efd35f9d9aa1a5e6bc22e379d2ff0afa2fc693887bdb607238"}
Oct 30 17:49:35.413123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:35.413094    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:35.589348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:35.589289    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:36.122750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:36.122711    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 17:49:36.265904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:36.265868    5534 generic.go:356] "Generic (PLEG): container finished" podID="257ca6b6-54bb-459a-a90d-2648bf6001bd" containerID="ffbc5f0f6df80b48fc46f099203976448a34b5f85e9382d08390c7bf8f754aa6" exitCode=0
Oct 30 17:49:36.266308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:36.265950    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4" event={"ID":"257ca6b6-54bb-459a-a90d-2648bf6001bd","Type":"ContainerDied","Data":"ffbc5f0f6df80b48fc46f099203976448a34b5f85e9382d08390c7bf8f754aa6"}
Oct 30 17:49:37.268961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.268925    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:37.515841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.515812    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:37.629309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.629256    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca\") pod \"257ca6b6-54bb-459a-a90d-2648bf6001bd\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") "
Oct 30 17:49:37.629514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.629496    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca\") pod \"257ca6b6-54bb-459a-a90d-2648bf6001bd\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") "
Oct 30 17:49:37.629832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.629787    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf\" (UniqueName: \"kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf\") pod \"257ca6b6-54bb-459a-a90d-2648bf6001bd\" (UID: \"257ca6b6-54bb-459a-a90d-2648bf6001bd\") "
Oct 30 17:49:37.629988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.629906    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca" pod "257ca6b6-54bb-459a-a90d-2648bf6001bd" (UID: "257ca6b6-54bb-459a-a90d-2648bf6001bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:37.630595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.630570    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:37.631938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.631912    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca" pod "257ca6b6-54bb-459a-a90d-2648bf6001bd" (UID: "257ca6b6-54bb-459a-a90d-2648bf6001bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:49:37.632692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.632629    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf" pod "257ca6b6-54bb-459a-a90d-2648bf6001bd" (UID: "257ca6b6-54bb-459a-a90d-2648bf6001bd"). InnerVolumeSpecName "kube-api-access-wr9vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:37.732577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.732548    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9vf\" (UniqueName: \"kubernetes.io/projected/257ca6b6-54bb-459a-a90d-2648bf6001bd-kube-api-access-wr9vf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:37.732577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:37.732575    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ca6b6-54bb-459a-a90d-2648bf6001bd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:38.293998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:38.293966    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"
Oct 30 17:49:38.293998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:38.293981    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4" event={"ID":"257ca6b6-54bb-459a-a90d-2648bf6001bd","Type":"ContainerDied","Data":"fa6ac96c618ee0efd35f9d9aa1a5e6bc22e379d2ff0afa2fc693887bdb607238"}
Oct 30 17:49:38.294420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:38.294033    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa6ac96c618ee0efd35f9d9aa1a5e6bc22e379d2ff0afa2fc693887bdb607238"
Oct 30 17:49:38.890041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:38.890012    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:40.716778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:40.716739    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:42.394771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:42.394735    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:43.071411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:43.071372    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:44.180733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:44.180461    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:44.916896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:44.916854    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"]
Oct 30 17:49:44.953154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:44.953124    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"]
Oct 30 17:49:45.047992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:45.047600    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:45.170890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:45.170806    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:45.252612 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:45.252577    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:49:46.221493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:46.221465    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:47.912910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:47.912872    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:49.310407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:49.310356    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:49.918811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:49.918761    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn" podUID="67499af8-ea4a-4461-a5f8-e58d0354893f" containerName="payload-processing" containerID="cri-o://0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76" gracePeriod=25
Oct 30 17:49:49.953601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:49.953556    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerName="payload-pre-processing" containerID="cri-o://abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06" gracePeriod=25
Oct 30 17:49:50.041617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.041588    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:50.223183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.223152    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:50.231499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.231454    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:49:50.244448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.244426    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"]
Oct 30 17:49:50.258082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.258058    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-874b83-wkbq4"]
Oct 30 17:49:50.336768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.330794    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume\") pod \"67499af8-ea4a-4461-a5f8-e58d0354893f\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") "
Oct 30 17:49:50.336768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.331098    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw\" (UniqueName: \"kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw\") pod \"67499af8-ea4a-4461-a5f8-e58d0354893f\" (UID: \"67499af8-ea4a-4461-a5f8-e58d0354893f\") "
Oct 30 17:49:50.336768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.334960    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume" pod "67499af8-ea4a-4461-a5f8-e58d0354893f" (UID: "67499af8-ea4a-4461-a5f8-e58d0354893f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:50.337460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.337192    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw" pod "67499af8-ea4a-4461-a5f8-e58d0354893f" (UID: "67499af8-ea4a-4461-a5f8-e58d0354893f"). InnerVolumeSpecName "kube-api-access-kscpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:50.341942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.341860    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:50.385859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.385826    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257ca6b6-54bb-459a-a90d-2648bf6001bd" path="/var/lib/kubelet/pods/257ca6b6-54bb-459a-a90d-2648bf6001bd/volumes"
Oct 30 17:49:50.399207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.399177    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"
Oct 30 17:49:50.399391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.399201    5534 generic.go:356] "Generic (PLEG): container finished" podID="67499af8-ea4a-4461-a5f8-e58d0354893f" containerID="0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76" exitCode=0
Oct 30 17:49:50.399391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.399258    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn" event={"ID":"67499af8-ea4a-4461-a5f8-e58d0354893f","Type":"ContainerDied","Data":"0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76"}
Oct 30 17:49:50.399391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.399314    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn" event={"ID":"67499af8-ea4a-4461-a5f8-e58d0354893f","Type":"ContainerDied","Data":"adbacfc40d44c7cef87c2edf38f0fd71aad8ee7e5c1e414bb3a9fb305246a3c2"}
Oct 30 17:49:50.399788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.399403    5534 scope.go:122] "RemoveContainer" containerID="0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76"
Oct 30 17:49:50.414339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.414318    5534 scope.go:122] "RemoveContainer" containerID="0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76"
Oct 30 17:49:50.414702 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:50.414617    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76\": container with ID starting with 0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76 not found: ID does not exist" containerID="0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76"
Oct 30 17:49:50.414814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.414715    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76"} err="failed to get container status \"0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76\": rpc error: code = NotFound desc = could not find container \"0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76\": container with ID starting with 0f22fbbf9a098ce8d680543c3a588f28e02c90dc666f04c8776918fbffd46b76 not found: ID does not exist"
Oct 30 17:49:50.436018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.435992    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67499af8-ea4a-4461-a5f8-e58d0354893f-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:50.436018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.436019    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscpw\" (UniqueName: \"kubernetes.io/projected/67499af8-ea4a-4461-a5f8-e58d0354893f-kube-api-access-kscpw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:50.460947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.460868    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"]
Oct 30 17:49:50.464845 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.464819    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-874b83-56655fc968-p55vn"]
Oct 30 17:49:50.795143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:50.795057    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:51.732394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:51.732342    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:52.169218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:52.169189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:52.375229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:52.375190    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67499af8-ea4a-4461-a5f8-e58d0354893f" path="/var/lib/kubelet/pods/67499af8-ea4a-4461-a5f8-e58d0354893f/volumes"
Oct 30 17:49:53.799908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:53.799880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:54.358388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.358348    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"]
Oct 30 17:49:54.359226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.359187    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" podUID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" containerName="discovery" containerID="cri-o://088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17" gracePeriod=30
Oct 30 17:49:54.676851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.676827    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:49:54.713932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.713904    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.714095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.713956    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.714095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.714064    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.714095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.714089    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.714264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.714120    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.714325 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.714284    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.714505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.714475    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:49:54.714618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.714592    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig\") pod \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\" (UID: \"20e977fe-38c6-4e11-8ada-3f55a8023fd3\") "
Oct 30 17:49:54.715815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.715626    5534 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-ca-configmap\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:54.716858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.716833    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "kube-api-access-csf55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:54.716988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.716954    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:49:54.717122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.717093    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:49:54.717197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.717123    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:49:54.717446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.717413    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:49:54.717495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.717428    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts" pod "20e977fe-38c6-4e11-8ada-3f55a8023fd3" (UID: "20e977fe-38c6-4e11-8ada-3f55a8023fd3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:49:54.817442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.817406    5534 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e977fe-38c6-4e11-8ada-3f55a8023fd3-local-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:54.817442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.817436    5534 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-cacerts\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:54.817442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.817446    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csf55\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-kube-api-access-csf55\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:54.817941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.817456    5534 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-kubeconfig\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:54.817941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.817466    5534 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-csr-dns-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:54.817941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:54.817476    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e977fe-38c6-4e11-8ada-3f55a8023fd3-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:49:55.449215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.449181    5534 generic.go:356] "Generic (PLEG): container finished" podID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" containerID="088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17" exitCode=0
Oct 30 17:49:55.449379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.449236    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"
Oct 30 17:49:55.449379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.449269    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" event={"ID":"20e977fe-38c6-4e11-8ada-3f55a8023fd3","Type":"ContainerDied","Data":"088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17"}
Oct 30 17:49:55.449379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.449324    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4" event={"ID":"20e977fe-38c6-4e11-8ada-3f55a8023fd3","Type":"ContainerDied","Data":"1c7fa622add4d6ee1f64f0435aeb5d8cccd95fd07f9e401c63cd912437545335"}
Oct 30 17:49:55.449379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.449348    5534 scope.go:122] "RemoveContainer" containerID="088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17"
Oct 30 17:49:55.463540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.463520    5534 scope.go:122] "RemoveContainer" containerID="088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17"
Oct 30 17:49:55.463884 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:49:55.463859    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17\": container with ID starting with 088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17 not found: ID does not exist" containerID="088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17"
Oct 30 17:49:55.463972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.463897    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17"} err="failed to get container status \"088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17\": rpc error: code = NotFound desc = could not find container \"088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17\": container with ID starting with 088af35229133faa3478bdc72c98e6fd0c321e446b2ad9ecc4a97ac128c9cf17 not found: ID does not exist"
Oct 30 17:49:55.504058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.504016    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"]
Oct 30 17:49:55.505037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.505001    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:55.516997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:55.516972    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t66m4"]
Oct 30 17:49:56.374437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:56.374391    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" path="/var/lib/kubelet/pods/20e977fe-38c6-4e11-8ada-3f55a8023fd3/volumes"
Oct 30 17:49:56.720393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:56.720295    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:49:57.275086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:57.275042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:57.591425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:57.591393    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:57.709883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:57.709843    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:57.963756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:57.962028    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"]
Oct 30 17:49:57.963756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:57.962379    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" containerID="cri-o://76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76" gracePeriod=30
Oct 30 17:49:57.967668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:57.965295    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:58.483009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.482972    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"]
Oct 30 17:49:58.483786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483767    5534 cpu_manager.go:414] "containerMap: removing container" podUID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" containerName="discovery"
Oct 30 17:49:58.483786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483785    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" containerName="discovery"
Oct 30 17:49:58.483945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483839    5534 cpu_manager.go:414] "containerMap: removing container" podUID="67499af8-ea4a-4461-a5f8-e58d0354893f" containerName="payload-processing"
Oct 30 17:49:58.483945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483844    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67499af8-ea4a-4461-a5f8-e58d0354893f" containerName="payload-processing"
Oct 30 17:49:58.483945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483859    5534 cpu_manager.go:414] "containerMap: removing container" podUID="257ca6b6-54bb-459a-a90d-2648bf6001bd" containerName="revoke-keys"
Oct 30 17:49:58.483945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483864    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257ca6b6-54bb-459a-a90d-2648bf6001bd" containerName="revoke-keys"
Oct 30 17:49:58.484170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483971    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="67499af8-ea4a-4461-a5f8-e58d0354893f" containerName="payload-processing"
Oct 30 17:49:58.484170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483980    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="257ca6b6-54bb-459a-a90d-2648bf6001bd" containerName="revoke-keys"
Oct 30 17:49:58.484170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.483987    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e977fe-38c6-4e11-8ada-3f55a8023fd3" containerName="discovery"
Oct 30 17:49:58.489592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.489562    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.509043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.509019    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"]
Oct 30 17:49:58.563689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.562692    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:49:58.563689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.562883    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:49:58.591995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.591958    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.592512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.592130    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8fc\" (UniqueName: \"kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.592512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.592260    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.693518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.693468    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.693765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.693634    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.693765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.693761    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8fc\" (UniqueName: \"kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.694563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.694539    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.696478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.696459    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.717600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.717570    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8fc\" (UniqueName: \"kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc\") pod \"maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:58.805681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:58.805584    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:49:59.096919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:59.096887    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:49:59.120512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:59.120474    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"]
Oct 30 17:49:59.121241 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:49:59.121215    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccd1839_7e05_42d2_b4b1_7d6044c6d47d.slice/crio-e70f549258432f2b7ad26271dc9d85fd6374728aaf64058b1cfb4fc60de99062 WatchSource:0}: Error finding container e70f549258432f2b7ad26271dc9d85fd6374728aaf64058b1cfb4fc60de99062: Status 404 returned error can't find the container with id e70f549258432f2b7ad26271dc9d85fd6374728aaf64058b1cfb4fc60de99062
Oct 30 17:49:59.490162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:49:59.490123    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf" event={"ID":"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d","Type":"ContainerStarted","Data":"e70f549258432f2b7ad26271dc9d85fd6374728aaf64058b1cfb4fc60de99062"}
Oct 30 17:50:00.078139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:00.078105    5534 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:50:00.078518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:00.078172    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:50:00.502496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:00.502457    5534 generic.go:356] "Generic (PLEG): container finished" podID="9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" containerID="67d4c7ae0005eb9629dc523ccae492bffd625ce2063e512104041257d97b8dfd" exitCode=0
Oct 30 17:50:00.502743 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:00.502537    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf" event={"ID":"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d","Type":"ContainerDied","Data":"67d4c7ae0005eb9629dc523ccae492bffd625ce2063e512104041257d97b8dfd"}
Oct 30 17:50:01.098321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.098266    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:01.213488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.213457    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:01.417885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.417804    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"]
Oct 30 17:50:01.418065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.418040    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" containerID="cri-o://c91a25f3f6af478d17dd47185d3246bd3e970dee697dcfa6c3c6616a2028ee43" gracePeriod=30
Oct 30 17:50:01.721250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.721224    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:50:01.846437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.846398    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc\" (UniqueName: \"kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc\") pod \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") "
Oct 30 17:50:01.846674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.846453    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca\") pod \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") "
Oct 30 17:50:01.846755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.846717    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca\") pod \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\" (UID: \"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d\") "
Oct 30 17:50:01.847112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.847083    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca" pod "9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" (UID: "9ccd1839-7e05-42d2-b4b1-7d6044c6d47d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:01.847864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.847841    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:01.849302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.849278    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca" pod "9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" (UID: "9ccd1839-7e05-42d2-b4b1-7d6044c6d47d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:50:01.850744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.850719    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc" pod "9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" (UID: "9ccd1839-7e05-42d2-b4b1-7d6044c6d47d"). InnerVolumeSpecName "kube-api-access-mm8fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:01.948917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.948843    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm8fc\" (UniqueName: \"kubernetes.io/projected/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-kube-api-access-mm8fc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:01.948917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:01.948868    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:02.337699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.337583    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tbsxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:50:02.337699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.337667    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:50:02.532276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.532247    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"
Oct 30 17:50:02.532487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.532245    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf" event={"ID":"9ccd1839-7e05-42d2-b4b1-7d6044c6d47d","Type":"ContainerDied","Data":"e70f549258432f2b7ad26271dc9d85fd6374728aaf64058b1cfb4fc60de99062"}
Oct 30 17:50:02.532487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.532369    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e70f549258432f2b7ad26271dc9d85fd6374728aaf64058b1cfb4fc60de99062"
Oct 30 17:50:02.724176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.724136    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:02.881689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.881655    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"]
Oct 30 17:50:02.882797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.882771    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" containerName="revoke-keys"
Oct 30 17:50:02.882797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.882795    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" containerName="revoke-keys"
Oct 30 17:50:02.883037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.882966    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" containerName="revoke-keys"
Oct 30 17:50:02.887815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.887567    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:02.910493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.908182    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:50:02.917175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.914279    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"]
Oct 30 17:50:02.917175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.914448    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:50:02.966948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.966914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26sk\" (UniqueName: \"kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:02.967115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.967038    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:02.967165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:02.967114    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.004980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.004958    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:03.069147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.069110    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26sk\" (UniqueName: \"kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.069359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.069204    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.069359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.069287    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.070403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.070374    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.071982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.071959    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.092210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.092186    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26sk\" (UniqueName: \"kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk\") pod \"maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.243835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.243743    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:03.319058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.318810    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:50:03.510268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510187    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510272    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510345    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510430    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510474    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510505    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510560    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510677    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:03.510780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510767    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:03.511189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510794    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:03.511189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510819    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.511189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510869    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy\") pod \"76898535-a858-4ebb-8178-d0b97521ee76\" (UID: \"76898535-a858-4ebb-8178-d0b97521ee76\") "
Oct 30 17:50:03.511189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.510991    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:03.511549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.511518    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.511549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.511552    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.511549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.511567    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.511913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.511582    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.511913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.511479    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:03.513197 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.513168    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:03.513326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.513242    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:50:03.513424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.513407    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "kube-api-access-mdstz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:03.513674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.513620    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy" pod "76898535-a858-4ebb-8178-d0b97521ee76" (UID: "76898535-a858-4ebb-8178-d0b97521ee76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:03.543260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.543220    5534 generic.go:356] "Generic (PLEG): container finished" podID="76898535-a858-4ebb-8178-d0b97521ee76" containerID="76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76" exitCode=0
Oct 30 17:50:03.543409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.543290    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"
Oct 30 17:50:03.543409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.543309    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" event={"ID":"76898535-a858-4ebb-8178-d0b97521ee76","Type":"ContainerDied","Data":"76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76"}
Oct 30 17:50:03.543409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.543374    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n" event={"ID":"76898535-a858-4ebb-8178-d0b97521ee76","Type":"ContainerDied","Data":"8203016b86c3fdf8995385cef05db9053921105705c296dff4face0200152cdb"}
Oct 30 17:50:03.543409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.543399    5534 scope.go:122] "RemoveContainer" containerID="76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76"
Oct 30 17:50:03.555756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.555735    5534 scope.go:122] "RemoveContainer" containerID="76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76"
Oct 30 17:50:03.556056 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:03.556037    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76\": container with ID starting with 76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76 not found: ID does not exist" containerID="76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76"
Oct 30 17:50:03.556117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.556066    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76"} err="failed to get container status \"76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76\": rpc error: code = NotFound desc = could not find container \"76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76\": container with ID starting with 76ea62fa52d5107affbddd78cd4f6ad6ff29f6c65bcbfc3eefc1ce3620bebc76 not found: ID does not exist"
Oct 30 17:50:03.583554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.583514    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"]
Oct 30 17:50:03.584177 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:50:03.584149    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42ecd87_829b_42d6_8c95_7a1dded5110d.slice/crio-fe8e35d76255864e756828d194829768bca4c9f372d2b942457ad299f5231ea8 WatchSource:0}: Error finding container fe8e35d76255864e756828d194829768bca4c9f372d2b942457ad299f5231ea8: Status 404 returned error can't find the container with id fe8e35d76255864e756828d194829768bca4c9f372d2b942457ad299f5231ea8
Oct 30 17:50:03.599862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.599833    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"]
Oct 30 17:50:03.607284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.607256    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-fvd4n"]
Oct 30 17:50:03.613537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.613508    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdstz\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-kube-api-access-mdstz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.613685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.613542    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76898535-a858-4ebb-8178-d0b97521ee76-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.613685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.613555    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76898535-a858-4ebb-8178-d0b97521ee76-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.613685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.613567    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76898535-a858-4ebb-8178-d0b97521ee76-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:03.613685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:03.613579    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76898535-a858-4ebb-8178-d0b97521ee76-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:04.375279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:04.375242    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76898535-a858-4ebb-8178-d0b97521ee76" path="/var/lib/kubelet/pods/76898535-a858-4ebb-8178-d0b97521ee76/volumes"
Oct 30 17:50:04.555223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:04.555180    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5" event={"ID":"f42ecd87-829b-42d6-8c95-7a1dded5110d","Type":"ContainerStarted","Data":"4849b47599ce15dd05f5bf8293b7f240dd412a3a23125a981ab82bb38c149211"}
Oct 30 17:50:04.555223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:04.555241    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5" event={"ID":"f42ecd87-829b-42d6-8c95-7a1dded5110d","Type":"ContainerStarted","Data":"fe8e35d76255864e756828d194829768bca4c9f372d2b942457ad299f5231ea8"}
Oct 30 17:50:04.608815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:04.608750    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5" podStartSLOduration=2.608734192 podStartE2EDuration="2.608734192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:04.594834666 +0000 UTC m=+2210.356878480" watchObservedRunningTime="2026-10-30 17:50:04.608734192 +0000 UTC m=+2210.370778010"
Oct 30 17:50:04.926707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:04.926666    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:05.443818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:05.443787    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:05.570303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:05.570263    5534 generic.go:356] "Generic (PLEG): container finished" podID="f42ecd87-829b-42d6-8c95-7a1dded5110d" containerID="4849b47599ce15dd05f5bf8293b7f240dd412a3a23125a981ab82bb38c149211" exitCode=0
Oct 30 17:50:05.570855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:05.570330    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5" event={"ID":"f42ecd87-829b-42d6-8c95-7a1dded5110d","Type":"ContainerDied","Data":"4849b47599ce15dd05f5bf8293b7f240dd412a3a23125a981ab82bb38c149211"}
Oct 30 17:50:06.583030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.582994    5534 generic.go:356] "Generic (PLEG): container finished" podID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerID="c91a25f3f6af478d17dd47185d3246bd3e970dee697dcfa6c3c6616a2028ee43" exitCode=0
Oct 30 17:50:06.583523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.583403    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" event={"ID":"2448eac0-836f-4471-ae1a-f9fa9c7c45ff","Type":"ContainerDied","Data":"c91a25f3f6af478d17dd47185d3246bd3e970dee697dcfa6c3c6616a2028ee43"}
Oct 30 17:50:06.724053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.723969    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:50:06.787447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.780136    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:50:06.832283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.832249    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:06.842278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.842244    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:06.874349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874314    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.874575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874375    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca\") pod \"f42ecd87-829b-42d6-8c95-7a1dded5110d\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") "
Oct 30 17:50:06.874575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874409    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk\" (UniqueName: \"kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk\") pod \"f42ecd87-829b-42d6-8c95-7a1dded5110d\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") "
Oct 30 17:50:06.874575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874433    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca\") pod \"f42ecd87-829b-42d6-8c95-7a1dded5110d\" (UID: \"f42ecd87-829b-42d6-8c95-7a1dded5110d\") "
Oct 30 17:50:06.874575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874560    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.874831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874604    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.874831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874696    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.874831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874767    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.874991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874905    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.874991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.874981    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.875094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.875049    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.875152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.875091    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert\") pod \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\" (UID: \"2448eac0-836f-4471-ae1a-f9fa9c7c45ff\") "
Oct 30 17:50:06.875724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.875537    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:06.875926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.875896    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:06.876129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.876103    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:06.876262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.876240    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.876351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.876270    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.876718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.876524    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:06.876918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.876865    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:06.877206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.877183    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca" pod "f42ecd87-829b-42d6-8c95-7a1dded5110d" (UID: "f42ecd87-829b-42d6-8c95-7a1dded5110d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:06.879235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.879199    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "kube-api-access-pglpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:06.880251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.880225    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:06.880469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.880449    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:50:06.881850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.881826    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token" pod "2448eac0-836f-4471-ae1a-f9fa9c7c45ff" (UID: "2448eac0-836f-4471-ae1a-f9fa9c7c45ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:06.881978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.881932    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk" pod "f42ecd87-829b-42d6-8c95-7a1dded5110d" (UID: "f42ecd87-829b-42d6-8c95-7a1dded5110d"). InnerVolumeSpecName "kube-api-access-r26sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:06.883817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.883607    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca" pod "f42ecd87-829b-42d6-8c95-7a1dded5110d" (UID: "f42ecd87-829b-42d6-8c95-7a1dded5110d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:50:06.942936 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.942844    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978060    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978092    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978101    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978112    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978121    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978137    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglpx\" (UniqueName: \"kubernetes.io/projected/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-kube-api-access-pglpx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978146    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2448eac0-836f-4471-ae1a-f9fa9c7c45ff-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978158    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978170    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26sk\" (UniqueName: \"kubernetes.io/projected/f42ecd87-829b-42d6-8c95-7a1dded5110d-kube-api-access-r26sk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:06.978169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:06.978179    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ecd87-829b-42d6-8c95-7a1dded5110d-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:07.600725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.599938    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"
Oct 30 17:50:07.600725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.600007    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh" event={"ID":"2448eac0-836f-4471-ae1a-f9fa9c7c45ff","Type":"ContainerDied","Data":"9b97cfa9f4b319c3006280bd7174cb4a7ebc0f29b12f91a46fa5c3fd83d0f9a0"}
Oct 30 17:50:07.600725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.600087    5534 scope.go:122] "RemoveContainer" containerID="c91a25f3f6af478d17dd47185d3246bd3e970dee697dcfa6c3c6616a2028ee43"
Oct 30 17:50:07.608155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.607593    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5" event={"ID":"f42ecd87-829b-42d6-8c95-7a1dded5110d","Type":"ContainerDied","Data":"fe8e35d76255864e756828d194829768bca4c9f372d2b942457ad299f5231ea8"}
Oct 30 17:50:07.608155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.607773    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"
Oct 30 17:50:07.608155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.607876    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8e35d76255864e756828d194829768bca4c9f372d2b942457ad299f5231ea8"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.613505    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"]
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614386    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f42ecd87-829b-42d6-8c95-7a1dded5110d" containerName="revoke-keys"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614401    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42ecd87-829b-42d6-8c95-7a1dded5110d" containerName="revoke-keys"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614417    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614422    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614432    5534 cpu_manager.go:414] "containerMap: removing container" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614438    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614601    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42ecd87-829b-42d6-8c95-7a1dded5110d" containerName="revoke-keys"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614615    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" containerName="istio-proxy"
Oct 30 17:50:07.615736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.614625    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="76898535-a858-4ebb-8178-d0b97521ee76" containerName="istio-proxy"
Oct 30 17:50:07.655948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.651968    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.667780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.667745    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4a271c\"/\"test-model-4a271c-kserve-self-signed-certs\""
Oct 30 17:50:07.668011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.667988    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4a271c\"/\"kube-root-ca.crt\""
Oct 30 17:50:07.668234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.668215    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4a271c\"/\"openshift-service-ca.crt\""
Oct 30 17:50:07.669496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.669472    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4a271c\"/\"default-dockercfg-l2qrb\""
Oct 30 17:50:07.670457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.670432    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:07.687203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.687143    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"]
Oct 30 17:50:07.690563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.690517    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.690804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.690626    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.690804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.690687    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.692435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.690971    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.692435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.692195    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dz6\" (UniqueName: \"kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.692435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.692421    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.698833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.698784    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"]
Oct 30 17:50:07.703442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.703347    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tbsxh"]
Oct 30 17:50:07.793534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.793497    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.793773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.793566    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.793773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.793593    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.793773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.793751    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.793773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.793771    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dz6\" (UniqueName: \"kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.794022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.793853    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.794085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.794049    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.794085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.794064    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.794206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.794137    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.796455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.796431    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.796792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.796776    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.817714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.817680    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dz6\" (UniqueName: \"kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6\") pod \"test-model-4a271c-kserve-6c8f689955-nlfgx\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") " pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:07.980739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:07.980630    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:08.305181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.305147    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"]
Oct 30 17:50:08.305396 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:50:08.305369    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod473abcc0_5027_4e5a_8c7e_d8f46645f1ad.slice/crio-5e8f626033733b19a44562e926ee491e453eb7349b3fbd4c3c233e00b1f7c98d WatchSource:0}: Error finding container 5e8f626033733b19a44562e926ee491e453eb7349b3fbd4c3c233e00b1f7c98d: Status 404 returned error can't find the container with id 5e8f626033733b19a44562e926ee491e453eb7349b3fbd4c3c233e00b1f7c98d
Oct 30 17:50:08.375104 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.375067    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2448eac0-836f-4471-ae1a-f9fa9c7c45ff" path="/var/lib/kubelet/pods/2448eac0-836f-4471-ae1a-f9fa9c7c45ff/volumes"
Oct 30 17:50:08.375489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.375473    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"]
Oct 30 17:50:08.382240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.382209    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.392822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.391712    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"]
Oct 30 17:50:08.400402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.400358    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-ce0480-serving-cert\""
Oct 30 17:50:08.472780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.472743    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:08.512939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.512888    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.513123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.513012    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.513432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.513410    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxxz\" (UniqueName: \"kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.513493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.513459    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.587832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.587721    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:08.614765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.614726    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxxz\" (UniqueName: \"kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.614765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.614780    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.615406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.614829    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.615406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.614864    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.615842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.615792    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.618608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.618577    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.622044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.622001    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.624383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.624314    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" event={"ID":"473abcc0-5027-4e5a-8c7e-d8f46645f1ad","Type":"ContainerStarted","Data":"a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968"}
Oct 30 17:50:08.624383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.624366    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" event={"ID":"473abcc0-5027-4e5a-8c7e-d8f46645f1ad","Type":"ContainerStarted","Data":"5e8f626033733b19a44562e926ee491e453eb7349b3fbd4c3c233e00b1f7c98d"}
Oct 30 17:50:08.641397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.641353    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxxz\" (UniqueName: \"kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz\") pod \"maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.673624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.673566    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" podStartSLOduration=1.673550928 podStartE2EDuration="1.673550928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:08.671093011 +0000 UTC m=+2214.433136826" watchObservedRunningTime="2026-10-30 17:50:08.673550928 +0000 UTC m=+2214.435594740"
Oct 30 17:50:08.706658 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.706609    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:08.731888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.731853    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:08.981345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:08.981297    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:09.086157 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:50:09.086117    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766c8b4e_6c73_4b5e_a920_178787f6e917.slice/crio-2cc274d283df15dcd0a50c8ff4dc00770555d4e4993b8d0baaed14eb45ca03c2 WatchSource:0}: Error finding container 2cc274d283df15dcd0a50c8ff4dc00770555d4e4993b8d0baaed14eb45ca03c2: Status 404 returned error can't find the container with id 2cc274d283df15dcd0a50c8ff4dc00770555d4e4993b8d0baaed14eb45ca03c2
Oct 30 17:50:09.087570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.087542    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"]
Oct 30 17:50:09.145367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.145262    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"]
Oct 30 17:50:09.292537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.292349    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:09.410125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.410027    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:09.642619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.642564    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n" event={"ID":"766c8b4e-6c73-4b5e-a920-178787f6e917","Type":"ContainerStarted","Data":"ed01f3ec2e6196111d1e023104a67511332f601508af6d448c200a0bc6601a21"}
Oct 30 17:50:09.643132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.642665    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n" event={"ID":"766c8b4e-6c73-4b5e-a920-178787f6e917","Type":"ContainerStarted","Data":"2cc274d283df15dcd0a50c8ff4dc00770555d4e4993b8d0baaed14eb45ca03c2"}
Oct 30 17:50:09.643132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.642787    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:09.689844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:09.689719    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n" podStartSLOduration=1.346769601 podStartE2EDuration="1.6896987s" totalImagesPullingTime="342.929099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:09.687151258 +0000 UTC m=+2215.449195073" watchObservedRunningTime="2026-10-30 17:50:09.6896987 +0000 UTC m=+2215.451742513"
Oct 30 17:50:10.304232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.304199    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:50:10.362331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.362288    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt\" (UniqueName: \"kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt\") pod \"ba263fbd-edcd-4193-9baf-f0629b2bae42\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") "
Oct 30 17:50:10.362679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.362636    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume\") pod \"ba263fbd-edcd-4193-9baf-f0629b2bae42\" (UID: \"ba263fbd-edcd-4193-9baf-f0629b2bae42\") "
Oct 30 17:50:10.363032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.363006    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume" pod "ba263fbd-edcd-4193-9baf-f0629b2bae42" (UID: "ba263fbd-edcd-4193-9baf-f0629b2bae42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:10.363437 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.363411    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba263fbd-edcd-4193-9baf-f0629b2bae42-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:10.366870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.366835    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt" pod "ba263fbd-edcd-4193-9baf-f0629b2bae42" (UID: "ba263fbd-edcd-4193-9baf-f0629b2bae42"). InnerVolumeSpecName "kube-api-access-gxzlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:10.464529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.464491    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxzlt\" (UniqueName: \"kubernetes.io/projected/ba263fbd-edcd-4193-9baf-f0629b2bae42-kube-api-access-gxzlt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:10.512954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.512916    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:10.652547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.652514    5534 generic.go:356] "Generic (PLEG): container finished" podID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerID="abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06" exitCode=0
Oct 30 17:50:10.653030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.652568    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"
Oct 30 17:50:10.653030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.652602    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" event={"ID":"ba263fbd-edcd-4193-9baf-f0629b2bae42","Type":"ContainerDied","Data":"abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06"}
Oct 30 17:50:10.653030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.652682    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c" event={"ID":"ba263fbd-edcd-4193-9baf-f0629b2bae42","Type":"ContainerDied","Data":"3ff9ff20491082e05f4caa17395ef87431dff579db5f0851724951c5110eac8c"}
Oct 30 17:50:10.653030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.652703    5534 scope.go:122] "RemoveContainer" containerID="abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06"
Oct 30 17:50:10.668874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.668846    5534 scope.go:122] "RemoveContainer" containerID="abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06"
Oct 30 17:50:10.669296 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:10.669273    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06\": container with ID starting with abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06 not found: ID does not exist" containerID="abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06"
Oct 30 17:50:10.669421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.669303    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06"} err="failed to get container status \"abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06\": rpc error: code = NotFound desc = could not find container \"abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06\": container with ID starting with abc8e68cf7c698956f111eb9ccd491ef875b3a7d12a4fd008192a6630d800f06 not found: ID does not exist"
Oct 30 17:50:10.706008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.705972    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"]
Oct 30 17:50:10.710089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:10.710058    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-874b83-75f7cfdd8c-hdk6c"]
Oct 30 17:50:12.200004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:12.199960    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:12.372695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:12.372633    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" path="/var/lib/kubelet/pods/ba263fbd-edcd-4193-9baf-f0629b2bae42/volumes"
Oct 30 17:50:13.339022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:13.338983    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"]
Oct 30 17:50:13.369927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:13.369891    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"]
Oct 30 17:50:13.473958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:13.473926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:13.595945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:13.595909    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:13.902002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:13.901907    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:14.146411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.146363    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg" podUID="ea87737e-0ada-43ac-8950-bf844b867894" containerName="payload-processing" containerID="cri-o://9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35" gracePeriod=25
Oct 30 17:50:14.494787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.494746    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:50:14.635288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.635256    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc\" (UniqueName: \"kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc\") pod \"ea87737e-0ada-43ac-8950-bf844b867894\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") "
Oct 30 17:50:14.635499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.635365    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume\") pod \"ea87737e-0ada-43ac-8950-bf844b867894\" (UID: \"ea87737e-0ada-43ac-8950-bf844b867894\") "
Oct 30 17:50:14.635781 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.635753    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume" pod "ea87737e-0ada-43ac-8950-bf844b867894" (UID: "ea87737e-0ada-43ac-8950-bf844b867894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:14.635927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.635911    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea87737e-0ada-43ac-8950-bf844b867894-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:14.637899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.637864    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc" pod "ea87737e-0ada-43ac-8950-bf844b867894" (UID: "ea87737e-0ada-43ac-8950-bf844b867894"). InnerVolumeSpecName "kube-api-access-972kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:14.702897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.702808    5534 generic.go:356] "Generic (PLEG): container finished" podID="ea87737e-0ada-43ac-8950-bf844b867894" containerID="9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35" exitCode=0
Oct 30 17:50:14.702897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.702882    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"
Oct 30 17:50:14.702897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.702888    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg" event={"ID":"ea87737e-0ada-43ac-8950-bf844b867894","Type":"ContainerDied","Data":"9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35"}
Oct 30 17:50:14.703129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.702955    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg" event={"ID":"ea87737e-0ada-43ac-8950-bf844b867894","Type":"ContainerDied","Data":"57a583e5683b0304208b7a95a56d0d18a4f9a09eaa2b62eafc58b1ec1c1e92c6"}
Oct 30 17:50:14.703129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.702980    5534 scope.go:122] "RemoveContainer" containerID="9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35"
Oct 30 17:50:14.716111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.716087    5534 scope.go:122] "RemoveContainer" containerID="9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35"
Oct 30 17:50:14.716478 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:14.716453    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35\": container with ID starting with 9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35 not found: ID does not exist" containerID="9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35"
Oct 30 17:50:14.716580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.716486    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35"} err="failed to get container status \"9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35\": rpc error: code = NotFound desc = could not find container \"9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35\": container with ID starting with 9767e414164e617ee29886f22b242ba0e86628767e31beaa6d509d5a6a49fd35 not found: ID does not exist"
Oct 30 17:50:14.737385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.737345    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972kc\" (UniqueName: \"kubernetes.io/projected/ea87737e-0ada-43ac-8950-bf844b867894-kube-api-access-972kc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:14.755492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.755443    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"]
Oct 30 17:50:14.778373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:14.778327    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee9ff3-6899847b98-m5wkg"]
Oct 30 17:50:15.040691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.040561    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"]
Oct 30 17:50:15.041384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.041367    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerName="payload-pre-processing"
Oct 30 17:50:15.041430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.041384    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerName="payload-pre-processing"
Oct 30 17:50:15.041430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.041406    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ea87737e-0ada-43ac-8950-bf844b867894" containerName="payload-processing"
Oct 30 17:50:15.041430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.041411    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87737e-0ada-43ac-8950-bf844b867894" containerName="payload-processing"
Oct 30 17:50:15.041557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.041548    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba263fbd-edcd-4193-9baf-f0629b2bae42" containerName="payload-pre-processing"
Oct 30 17:50:15.041598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.041561    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87737e-0ada-43ac-8950-bf844b867894" containerName="payload-processing"
Oct 30 17:50:15.046823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.046799    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.061589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.061551    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:50:15.061964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.061943    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:50:15.068087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.068048    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"]
Oct 30 17:50:15.143146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.143100    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.143370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.143336    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.143475 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.143442    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxr2f\" (UniqueName: \"kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.244915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.244876    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.245196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.245035    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.245196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.245110    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxr2f\" (UniqueName: \"kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.245788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.245765    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.247816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.247783    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.250452 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:15.250419    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:50:15.265345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.265293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:15.270117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.270081    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxr2f\" (UniqueName: \"kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f\") pod \"maas-api-revoke-sub-16e96046-p2v5c\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.365016 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.364975    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:15.682926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.682743    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"]
Oct 30 17:50:15.686226 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:50:15.686188    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea224c8_d022_4f0f_84a8_900f21d00483.slice/crio-0990bdeedf40c13a3f5fccb122e169abcbe74173e97375c13453e7c0a3f4eae2 WatchSource:0}: Error finding container 0990bdeedf40c13a3f5fccb122e169abcbe74173e97375c13453e7c0a3f4eae2: Status 404 returned error can't find the container with id 0990bdeedf40c13a3f5fccb122e169abcbe74173e97375c13453e7c0a3f4eae2
Oct 30 17:50:15.713035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.713001    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c" event={"ID":"5ea224c8-d022-4f0f-84a8-900f21d00483","Type":"ContainerStarted","Data":"0990bdeedf40c13a3f5fccb122e169abcbe74173e97375c13453e7c0a3f4eae2"}
Oct 30 17:50:15.827012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:15.826972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:16.251505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:16.251378    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:16.374577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:16.374544    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea87737e-0ada-43ac-8950-bf844b867894" path="/var/lib/kubelet/pods/ea87737e-0ada-43ac-8950-bf844b867894/volumes"
Oct 30 17:50:16.672664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:16.671825    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 17:50:16.725300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:16.725266    5534 generic.go:356] "Generic (PLEG): container finished" podID="5ea224c8-d022-4f0f-84a8-900f21d00483" containerID="b500ba887bad3feca63a91ba58ecf93ed23beaa6557c16e7ad3e140de3398f0e" exitCode=0
Oct 30 17:50:16.725838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:16.725339    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c" event={"ID":"5ea224c8-d022-4f0f-84a8-900f21d00483","Type":"ContainerDied","Data":"b500ba887bad3feca63a91ba58ecf93ed23beaa6557c16e7ad3e140de3398f0e"}
Oct 30 17:50:17.547714 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.547683    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:17.709205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.709174    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"]
Oct 30 17:50:17.715590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.715561    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.755051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.755006    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"]
Oct 30 17:50:17.757129 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.757055    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c596f1\"/\"default-dockercfg-bz5r9\""
Oct 30 17:50:17.757329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.757083    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c596f1\"/\"kube-root-ca.crt\""
Oct 30 17:50:17.757428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.757212    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c596f1\"/\"test-model-c596f1-kserve-self-signed-certs\""
Oct 30 17:50:17.757509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.757291    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c596f1\"/\"openshift-service-ca.crt\""
Oct 30 17:50:17.761761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.761691    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:17.788695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.788624    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjbh\" (UniqueName: \"kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.788901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.788815    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.788901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.788847    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.789069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.788978    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.789125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.789075    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.789203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.789185    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.890953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.890902    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjbh\" (UniqueName: \"kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.891134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.891052    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.891134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.891083    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.891287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.891202    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.891344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.891303    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.891444 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.891408    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.891930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.891901    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.892089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.892065    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.892972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.892925    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.907705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.897029    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.907705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.905287    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.916379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.916315    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjbh\" (UniqueName: \"kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh\") pod \"test-model-c596f1-kserve-5c759849bc-hmpsw\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") " pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:17.985039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.981865    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:17.988855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:17.988825    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:18.039236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.037977    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:18.066690 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.056158    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:18.099911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.099830    5534 scope.go:122] "RemoveContainer" containerID="69cb7acd6f5ba73592196f67184a46ee5fd146d6e294c5873dae34005e21fd01"
Oct 30 17:50:18.118748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.117208    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca\") pod \"5ea224c8-d022-4f0f-84a8-900f21d00483\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") "
Oct 30 17:50:18.118748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.117403    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca\") pod \"5ea224c8-d022-4f0f-84a8-900f21d00483\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") "
Oct 30 17:50:18.118748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.117536    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f\" (UniqueName: \"kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f\") pod \"5ea224c8-d022-4f0f-84a8-900f21d00483\" (UID: \"5ea224c8-d022-4f0f-84a8-900f21d00483\") "
Oct 30 17:50:18.118748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.117956    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca" pod "5ea224c8-d022-4f0f-84a8-900f21d00483" (UID: "5ea224c8-d022-4f0f-84a8-900f21d00483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:18.118748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.118386    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:18.128098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.122767    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca" pod "5ea224c8-d022-4f0f-84a8-900f21d00483" (UID: "5ea224c8-d022-4f0f-84a8-900f21d00483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:50:18.128098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.122939    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f" pod "5ea224c8-d022-4f0f-84a8-900f21d00483" (UID: "5ea224c8-d022-4f0f-84a8-900f21d00483"). InnerVolumeSpecName "kube-api-access-xxr2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:18.220041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.220011    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxr2f\" (UniqueName: \"kubernetes.io/projected/5ea224c8-d022-4f0f-84a8-900f21d00483-kube-api-access-xxr2f\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:18.220130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.220050    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea224c8-d022-4f0f-84a8-900f21d00483-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:18.340107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.340016    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerName="payload-processing" containerID="cri-o://e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68" gracePeriod=25
Oct 30 17:50:18.370712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.370537    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" podUID="9660209e-2cbf-4834-824b-005c96f6e624" containerName="payload-pre-processing" containerID="cri-o://f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128" gracePeriod=25
Oct 30 17:50:18.389756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.387580    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"]
Oct 30 17:50:18.776143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.776043    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"
Oct 30 17:50:18.776143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.776058    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c" event={"ID":"5ea224c8-d022-4f0f-84a8-900f21d00483","Type":"ContainerDied","Data":"0990bdeedf40c13a3f5fccb122e169abcbe74173e97375c13453e7c0a3f4eae2"}
Oct 30 17:50:18.776143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.776115    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0990bdeedf40c13a3f5fccb122e169abcbe74173e97375c13453e7c0a3f4eae2"
Oct 30 17:50:18.778561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.778529    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw" event={"ID":"256d788e-3a50-4a0d-87bc-95567f350332","Type":"ContainerStarted","Data":"fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75"}
Oct 30 17:50:18.778741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.778575    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw" event={"ID":"256d788e-3a50-4a0d-87bc-95567f350332","Type":"ContainerStarted","Data":"3fb24a4427f7e3704b4fd14d588b37a8effbbe50cb3ccf72ce76777851f4afe5"}
Oct 30 17:50:18.790686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.790658    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:50:18.823381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:18.823327    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw" podStartSLOduration=1.823310732 podStartE2EDuration="1.823310732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:18.81889866 +0000 UTC m=+2224.580942474" watchObservedRunningTime="2026-10-30 17:50:18.823310732 +0000 UTC m=+2224.585354569"
Oct 30 17:50:19.038891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:19.038803    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:19.323510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:19.323418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:21.064152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:21.064116    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:21.187575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:21.187490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:22.769495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:22.769454    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:24.395904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:24.395866    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:26.206470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:26.206434    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:26.720881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:26.720842    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" podUID="9660209e-2cbf-4834-824b-005c96f6e624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:50:27.454219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:27.454147    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:27.750254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:27.750132    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:50:28.043792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:28.043696    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:28.135671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:28.125999    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:28.366075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:28.366039    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:29.113843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:29.113811    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:50:30.397844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:30.397815    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:32.210300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:32.210259    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:33.804117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:33.804086    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:35.388615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:35.388576    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:36.721196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:36.721151    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" podUID="9660209e-2cbf-4834-824b-005c96f6e624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:50:37.181620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:37.181582    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:37.728774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:37.728732    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:50:38.640099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.639975    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:38.723653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.723604    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:50:38.771930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.771897    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:50:38.875736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.872463    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq\" (UniqueName: \"kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq\") pod \"9660209e-2cbf-4834-824b-005c96f6e624\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") "
Oct 30 17:50:38.875736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.872807    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume\") pod \"188e41bd-2374-4dc9-9ba5-824ada2f7647\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") "
Oct 30 17:50:38.876667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.876624    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume\") pod \"9660209e-2cbf-4834-824b-005c96f6e624\" (UID: \"9660209e-2cbf-4834-824b-005c96f6e624\") "
Oct 30 17:50:38.876830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.876726    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg\" (UniqueName: \"kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg\") pod \"188e41bd-2374-4dc9-9ba5-824ada2f7647\" (UID: \"188e41bd-2374-4dc9-9ba5-824ada2f7647\") "
Oct 30 17:50:38.877973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.876313    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume" pod "188e41bd-2374-4dc9-9ba5-824ada2f7647" (UID: "188e41bd-2374-4dc9-9ba5-824ada2f7647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:38.877973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.877924    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume" pod "9660209e-2cbf-4834-824b-005c96f6e624" (UID: "9660209e-2cbf-4834-824b-005c96f6e624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:38.899901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.896532    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg" pod "188e41bd-2374-4dc9-9ba5-824ada2f7647" (UID: "188e41bd-2374-4dc9-9ba5-824ada2f7647"). InnerVolumeSpecName "kube-api-access-gfrkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:38.900942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.900878    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq" pod "9660209e-2cbf-4834-824b-005c96f6e624" (UID: "9660209e-2cbf-4834-824b-005c96f6e624"). InnerVolumeSpecName "kube-api-access-tdqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:38.983031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.982990    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188e41bd-2374-4dc9-9ba5-824ada2f7647-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:38.983031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.983034    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9660209e-2cbf-4834-824b-005c96f6e624-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:38.983312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.983050    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfrkg\" (UniqueName: \"kubernetes.io/projected/188e41bd-2374-4dc9-9ba5-824ada2f7647-kube-api-access-gfrkg\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:38.983312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:38.983065    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqqq\" (UniqueName: \"kubernetes.io/projected/9660209e-2cbf-4834-824b-005c96f6e624-kube-api-access-tdqqq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:39.173844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.173747    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:39.210689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.210628    5534 generic.go:356] "Generic (PLEG): container finished" podID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerID="e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68" exitCode=0
Oct 30 17:50:39.211003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.210974    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" event={"ID":"188e41bd-2374-4dc9-9ba5-824ada2f7647","Type":"ContainerDied","Data":"e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68"}
Oct 30 17:50:39.211106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.211092    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d" event={"ID":"188e41bd-2374-4dc9-9ba5-824ada2f7647","Type":"ContainerDied","Data":"1f5a7b4e8c92b49589a08a8cefedec158d7453257b99daf7b83a5c07c4805604"}
Oct 30 17:50:39.211161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.211140    5534 scope.go:122] "RemoveContainer" containerID="e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68"
Oct 30 17:50:39.211487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.210908    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"
Oct 30 17:50:39.215748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.215717    5534 generic.go:356] "Generic (PLEG): container finished" podID="9660209e-2cbf-4834-824b-005c96f6e624" containerID="f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128" exitCode=0
Oct 30 17:50:39.216075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.216052    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" event={"ID":"9660209e-2cbf-4834-824b-005c96f6e624","Type":"ContainerDied","Data":"f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128"}
Oct 30 17:50:39.217195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.217171    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9" event={"ID":"9660209e-2cbf-4834-824b-005c96f6e624","Type":"ContainerDied","Data":"24a60a0eac081afb86be0d803085fa8d7b9070a0e8cf36690819a6a89ae75307"}
Oct 30 17:50:39.217425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.217410    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"
Oct 30 17:50:39.243960 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.243934    5534 scope.go:122] "RemoveContainer" containerID="e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68"
Oct 30 17:50:39.244368 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:39.244344    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68\": container with ID starting with e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68 not found: ID does not exist" containerID="e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68"
Oct 30 17:50:39.244473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.244376    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68"} err="failed to get container status \"e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68\": rpc error: code = NotFound desc = could not find container \"e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68\": container with ID starting with e7e9072e6b125d32f2ae98b5034f2298b1f56988bac107f4784a100c54079a68 not found: ID does not exist"
Oct 30 17:50:39.244473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.244395    5534 scope.go:122] "RemoveContainer" containerID="f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128"
Oct 30 17:50:39.256884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.256861    5534 scope.go:122] "RemoveContainer" containerID="f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128"
Oct 30 17:50:39.257350 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:39.257326    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128\": container with ID starting with f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128 not found: ID does not exist" containerID="f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128"
Oct 30 17:50:39.257435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.257360    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128"} err="failed to get container status \"f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128\": rpc error: code = NotFound desc = could not find container \"f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128\": container with ID starting with f254c04891c0bac8a191e5541c9b265abbf28f0ca5aba53707533f00c78ea128 not found: ID does not exist"
Oct 30 17:50:39.286181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.286150    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"]
Oct 30 17:50:39.295689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.295658    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:39.301466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.301433    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f8b48d-7666797cf-2d7p9"]
Oct 30 17:50:39.312393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.312357    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"]
Oct 30 17:50:39.319270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.319244    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8b48d-8xmr5"]
Oct 30 17:50:39.343106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.343074    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"]
Oct 30 17:50:39.349521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.349490    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8b48d-9ff5d88bd-gts5d"]
Oct 30 17:50:39.428396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.428286    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:39.810154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:39.810065    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:40.372694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:40.372656    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" path="/var/lib/kubelet/pods/188e41bd-2374-4dc9-9ba5-824ada2f7647/volumes"
Oct 30 17:50:40.373315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:40.373288    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9660209e-2cbf-4834-824b-005c96f6e624" path="/var/lib/kubelet/pods/9660209e-2cbf-4834-824b-005c96f6e624/volumes"
Oct 30 17:50:40.373810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:40.373792    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42ecd87-829b-42d6-8c95-7a1dded5110d" path="/var/lib/kubelet/pods/f42ecd87-829b-42d6-8c95-7a1dded5110d/volumes"
Oct 30 17:50:41.033697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:41.033623    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:41.922047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:41.922016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:42.040806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.040772    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:42.102358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.102311    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"]
Oct 30 17:50:42.143669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.131155    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee9ff3-qqssf"]
Oct 30 17:50:42.163667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.156760    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:42.374192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.374154    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccd1839-7e05-42d2-b4b1-7d6044c6d47d" path="/var/lib/kubelet/pods/9ccd1839-7e05-42d2-b4b1-7d6044c6d47d/volumes"
Oct 30 17:50:42.463071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.463035    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"]
Oct 30 17:50:42.464077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464055    5534 cpu_manager.go:414] "containerMap: removing container" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerName="payload-processing"
Oct 30 17:50:42.464077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464078    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerName="payload-processing"
Oct 30 17:50:42.464277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464120    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9660209e-2cbf-4834-824b-005c96f6e624" containerName="payload-pre-processing"
Oct 30 17:50:42.464277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464128    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9660209e-2cbf-4834-824b-005c96f6e624" containerName="payload-pre-processing"
Oct 30 17:50:42.464277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464182    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5ea224c8-d022-4f0f-84a8-900f21d00483" containerName="revoke-keys"
Oct 30 17:50:42.464277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464189    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea224c8-d022-4f0f-84a8-900f21d00483" containerName="revoke-keys"
Oct 30 17:50:42.464495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464348    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9660209e-2cbf-4834-824b-005c96f6e624" containerName="payload-pre-processing"
Oct 30 17:50:42.464495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464370    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea224c8-d022-4f0f-84a8-900f21d00483" containerName="revoke-keys"
Oct 30 17:50:42.464495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.464382    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="188e41bd-2374-4dc9-9ba5-824ada2f7647" containerName="payload-processing"
Oct 30 17:50:42.470201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.470178    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.485097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.485071    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:50:42.485336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.485313    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:50:42.487689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.487166    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"]
Oct 30 17:50:42.559939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.559900    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.560168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.560106    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksj7\" (UniqueName: \"kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.560168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.560161    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.616561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.616525    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:42.661253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.661171    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.661418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.661359    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksj7\" (UniqueName: \"kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.661498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.661418    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.662277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.662248    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.664023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.663995    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.685351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.685327    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksj7\" (UniqueName: \"kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7\") pod \"maas-api-revoke-sub-ecc4b213-849zb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:42.793435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:42.793403    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:43.109262 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:50:43.109223    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ed7c40_4e5f_4748_8735_d163d6360fcb.slice/crio-b28ac89cb33adeb20855020650de3fafa8490509d632753ff53de099226f3daa WatchSource:0}: Error finding container b28ac89cb33adeb20855020650de3fafa8490509d632753ff53de099226f3daa: Status 404 returned error can't find the container with id b28ac89cb33adeb20855020650de3fafa8490509d632753ff53de099226f3daa
Oct 30 17:50:43.109768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:43.109746    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"]
Oct 30 17:50:43.300944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:43.300797    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb" event={"ID":"88ed7c40-4e5f-4748-8735-d163d6360fcb","Type":"ContainerStarted","Data":"b28ac89cb33adeb20855020650de3fafa8490509d632753ff53de099226f3daa"}
Oct 30 17:50:43.857903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:43.857868    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:44.311208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:44.311169    5534 generic.go:356] "Generic (PLEG): container finished" podID="88ed7c40-4e5f-4748-8735-d163d6360fcb" containerID="2fd1a18da53c05a2f1dba17991c10babd48a26ada32090ee2684b8914052af2c" exitCode=0
Oct 30 17:50:44.311712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:44.311210    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb" event={"ID":"88ed7c40-4e5f-4748-8735-d163d6360fcb","Type":"ContainerDied","Data":"2fd1a18da53c05a2f1dba17991c10babd48a26ada32090ee2684b8914052af2c"}
Oct 30 17:50:45.246881 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:45.246839    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:50:45.490507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.490480    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:45.607224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.607190    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca\") pod \"88ed7c40-4e5f-4748-8735-d163d6360fcb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") "
Oct 30 17:50:45.607421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.607337    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7\" (UniqueName: \"kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7\") pod \"88ed7c40-4e5f-4748-8735-d163d6360fcb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") "
Oct 30 17:50:45.607421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.607416    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca\") pod \"88ed7c40-4e5f-4748-8735-d163d6360fcb\" (UID: \"88ed7c40-4e5f-4748-8735-d163d6360fcb\") "
Oct 30 17:50:45.607954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.607876    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca" pod "88ed7c40-4e5f-4748-8735-d163d6360fcb" (UID: "88ed7c40-4e5f-4748-8735-d163d6360fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:45.608142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.608046    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:45.609558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.609529    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca" pod "88ed7c40-4e5f-4748-8735-d163d6360fcb" (UID: "88ed7c40-4e5f-4748-8735-d163d6360fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:50:45.609729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.609628    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7" pod "88ed7c40-4e5f-4748-8735-d163d6360fcb" (UID: "88ed7c40-4e5f-4748-8735-d163d6360fcb"). InnerVolumeSpecName "kube-api-access-kksj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:45.673107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.673016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:45.709892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.709855    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ed7c40-4e5f-4748-8735-d163d6360fcb-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:45.709892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:45.709894    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kksj7\" (UniqueName: \"kubernetes.io/projected/88ed7c40-4e5f-4748-8735-d163d6360fcb-kube-api-access-kksj7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:46.276018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:46.275980    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:46.344976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:46.343558    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb" event={"ID":"88ed7c40-4e5f-4748-8735-d163d6360fcb","Type":"ContainerDied","Data":"b28ac89cb33adeb20855020650de3fafa8490509d632753ff53de099226f3daa"}
Oct 30 17:50:46.344976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:46.343713    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28ac89cb33adeb20855020650de3fafa8490509d632753ff53de099226f3daa"
Oct 30 17:50:46.344976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:46.343875    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"
Oct 30 17:50:46.406007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:46.405946    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:46.625846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:46.625813    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:47.051888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.051779    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:47.136240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.136206    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"]
Oct 30 17:50:47.136976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.136960    5534 cpu_manager.go:414] "containerMap: removing container" podUID="88ed7c40-4e5f-4748-8735-d163d6360fcb" containerName="revoke-keys"
Oct 30 17:50:47.136976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.136977    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ed7c40-4e5f-4748-8735-d163d6360fcb" containerName="revoke-keys"
Oct 30 17:50:47.137144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.137134    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ed7c40-4e5f-4748-8735-d163d6360fcb" containerName="revoke-keys"
Oct 30 17:50:47.141122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.141100    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.162452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.162420    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"]
Oct 30 17:50:47.192777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.192743    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:47.243431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.243391    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.243663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.243573    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zq48\" (UniqueName: \"kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.243750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.243731    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.251073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.251047    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:50:47.251216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.251106    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:50:47.345256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.345229    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.345451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.345338    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zq48\" (UniqueName: \"kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.345451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.345443    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.346017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.345990    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.348154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.348127    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.369558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.369517    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zq48\" (UniqueName: \"kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48\") pod \"maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.461302 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.461264    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:47.733565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.733534    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:47.796985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:47.796953    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"]
Oct 30 17:50:48.376580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:48.376494    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f" event={"ID":"b86ae4bd-e84f-4f91-b262-160ba2b36c66","Type":"ContainerStarted","Data":"4813d7ca5aa9542967c683dd01df3590502e318f73e9f91482351e726a54d774"}
Oct 30 17:50:48.376580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:48.376571    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f" event={"ID":"b86ae4bd-e84f-4f91-b262-160ba2b36c66","Type":"ContainerStarted","Data":"d42366fabeae2ca6a855132d011a1fd521f50740d398fa7c72c175f40c4be0ff"}
Oct 30 17:50:48.409383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:48.409321    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f" podStartSLOduration=1.409305353 podStartE2EDuration="1.409305353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:48.405254552 +0000 UTC m=+2254.167298368" watchObservedRunningTime="2026-10-30 17:50:48.409305353 +0000 UTC m=+2254.171349171"
Oct 30 17:50:49.234467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.234431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:49.350666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.350122    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:49.402712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.399581    5534 generic.go:356] "Generic (PLEG): container finished" podID="b86ae4bd-e84f-4f91-b262-160ba2b36c66" containerID="4813d7ca5aa9542967c683dd01df3590502e318f73e9f91482351e726a54d774" exitCode=0
Oct 30 17:50:49.402712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.399796    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f" event={"ID":"b86ae4bd-e84f-4f91-b262-160ba2b36c66","Type":"ContainerDied","Data":"4813d7ca5aa9542967c683dd01df3590502e318f73e9f91482351e726a54d774"}
Oct 30 17:50:49.559725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.558693    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"]
Oct 30 17:50:49.559725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.559351    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" containerID="cri-o://c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1" gracePeriod=30
Oct 30 17:50:49.611077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.611046    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:49.760666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:49.759241    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:50.596099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.596072    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:50.727738 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.727621    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca\") pod \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") "
Oct 30 17:50:50.727945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.727792    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca\") pod \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") "
Oct 30 17:50:50.728026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.727941    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48\" (UniqueName: \"kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48\") pod \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\" (UID: \"b86ae4bd-e84f-4f91-b262-160ba2b36c66\") "
Oct 30 17:50:50.728120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.728089    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca" pod "b86ae4bd-e84f-4f91-b262-160ba2b36c66" (UID: "b86ae4bd-e84f-4f91-b262-160ba2b36c66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:50.728510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.728486    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:50.730150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.730125    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca" pod "b86ae4bd-e84f-4f91-b262-160ba2b36c66" (UID: "b86ae4bd-e84f-4f91-b262-160ba2b36c66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:50:50.730369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.730345    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48" pod "b86ae4bd-e84f-4f91-b262-160ba2b36c66" (UID: "b86ae4bd-e84f-4f91-b262-160ba2b36c66"). InnerVolumeSpecName "kube-api-access-7zq48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:50.832671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.830091    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zq48\" (UniqueName: \"kubernetes.io/projected/b86ae4bd-e84f-4f91-b262-160ba2b36c66-kube-api-access-7zq48\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:50.832671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:50.830129    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86ae4bd-e84f-4f91-b262-160ba2b36c66-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:51.417680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.417637    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"
Oct 30 17:50:51.417874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.417681    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f" event={"ID":"b86ae4bd-e84f-4f91-b262-160ba2b36c66","Type":"ContainerDied","Data":"d42366fabeae2ca6a855132d011a1fd521f50740d398fa7c72c175f40c4be0ff"}
Oct 30 17:50:51.417874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.417732    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42366fabeae2ca6a855132d011a1fd521f50740d398fa7c72c175f40c4be0ff"
Oct 30 17:50:51.587916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.587874    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"]
Oct 30 17:50:51.588709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.588690    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b86ae4bd-e84f-4f91-b262-160ba2b36c66" containerName="revoke-keys"
Oct 30 17:50:51.588709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.588709    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86ae4bd-e84f-4f91-b262-160ba2b36c66" containerName="revoke-keys"
Oct 30 17:50:51.588920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.588907    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86ae4bd-e84f-4f91-b262-160ba2b36c66" containerName="revoke-keys"
Oct 30 17:50:51.592910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.592889    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.609484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.609440    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:50:51.609989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.609542    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:50:51.618427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.618395    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"]
Oct 30 17:50:51.640292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.640256    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:51.744075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.743975    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9mw\" (UniqueName: \"kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.744308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.744192    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.744447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.744426    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.845711 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.845662    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.845900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.845794    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9mw\" (UniqueName: \"kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.845900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.845864    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.846285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.846261    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.848503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.848478    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.868814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.868780    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9mw\" (UniqueName: \"kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw\") pod \"maas-api-revoke-sub-938023d5-pljm2\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:51.909862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:51.909828    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:52.243057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:52.243027    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"]
Oct 30 17:50:52.243725 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:50:52.243698    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102a1711_2402_4fb0_81ad_07eed42da980.slice/crio-5cff7d5aad87d60989b2bf59a0379234a4f1c1981970e772e72e7f21b9116442 WatchSource:0}: Error finding container 5cff7d5aad87d60989b2bf59a0379234a4f1c1981970e772e72e7f21b9116442: Status 404 returned error can't find the container with id 5cff7d5aad87d60989b2bf59a0379234a4f1c1981970e772e72e7f21b9116442
Oct 30 17:50:52.427092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:52.427056    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2" event={"ID":"102a1711-2402-4fb0-81ad-07eed42da980","Type":"ContainerStarted","Data":"5cff7d5aad87d60989b2bf59a0379234a4f1c1981970e772e72e7f21b9116442"}
Oct 30 17:50:53.253467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:53.253325    5534 patch_prober.go:29] interesting pod/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:50:53.253467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:53.253387    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:50:53.438729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:53.438696    5534 generic.go:356] "Generic (PLEG): container finished" podID="102a1711-2402-4fb0-81ad-07eed42da980" containerID="3602b9a0c9bd1bbc5aad4da060321145dd51872be995cc435af050e2123f4716" exitCode=0
Oct 30 17:50:53.438921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:53.438796    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2" event={"ID":"102a1711-2402-4fb0-81ad-07eed42da980","Type":"ContainerDied","Data":"3602b9a0c9bd1bbc5aad4da060321145dd51872be995cc435af050e2123f4716"}
Oct 30 17:50:53.443826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:53.443798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:54.705328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.705263    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.791526    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca\") pod \"102a1711-2402-4fb0-81ad-07eed42da980\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") "
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.791795    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw\" (UniqueName: \"kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw\") pod \"102a1711-2402-4fb0-81ad-07eed42da980\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") "
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.792774    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca\") pod \"102a1711-2402-4fb0-81ad-07eed42da980\" (UID: \"102a1711-2402-4fb0-81ad-07eed42da980\") "
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.798278    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca" pod "102a1711-2402-4fb0-81ad-07eed42da980" (UID: "102a1711-2402-4fb0-81ad-07eed42da980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.803045    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca" pod "102a1711-2402-4fb0-81ad-07eed42da980" (UID: "102a1711-2402-4fb0-81ad-07eed42da980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.803384    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:54.805736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.803416    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102a1711-2402-4fb0-81ad-07eed42da980-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:54.878674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.872970    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw" pod "102a1711-2402-4fb0-81ad-07eed42da980" (UID: "102a1711-2402-4fb0-81ad-07eed42da980"). InnerVolumeSpecName "kube-api-access-7c9mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:54.988904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:54.970958    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c9mw\" (UniqueName: \"kubernetes.io/projected/102a1711-2402-4fb0-81ad-07eed42da980-kube-api-access-7c9mw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.024009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.023982    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:50:55.173359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173326    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.173558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173373    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.173558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173429    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.173680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173592    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.173750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173706    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.173849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173831    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.173967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173950    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.174027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.173990    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.174066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174014    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:55.174066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174056    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket\") pod \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\" (UID: \"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10\") "
Oct 30 17:50:55.174399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174360    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:55.174399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174392    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:50:55.174566 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174544    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:55.174786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174766    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.174883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174790    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.174883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174803    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.174883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174816    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.174883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.174824    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:55.176785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.176758    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:50:55.176785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.176775    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:50:55.176943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.176918    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:55.177333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.177308    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2" pod "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" (UID: "2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10"). InnerVolumeSpecName "kube-api-access-x8qv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:50:55.241576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.241495    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:55.276290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.276249    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.276290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.276281    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.276290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.276294    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8qv2\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-kube-api-access-x8qv2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.276791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.276307    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.276791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.276320    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:50:55.461908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.461867    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerID="c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1" exitCode=0
Oct 30 17:50:55.462122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.461931    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"
Oct 30 17:50:55.462122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.461954    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" event={"ID":"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10","Type":"ContainerDied","Data":"c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1"}
Oct 30 17:50:55.462122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.462025    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4" event={"ID":"2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10","Type":"ContainerDied","Data":"11fc9da4e7603be5bf1d526c810675b8fc78a76b0d1ff714af7f2e18a503ba2e"}
Oct 30 17:50:55.462122 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.462056    5534 scope.go:122] "RemoveContainer" containerID="c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1"
Oct 30 17:50:55.463729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.463710    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"
Oct 30 17:50:55.464028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.463821    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2" event={"ID":"102a1711-2402-4fb0-81ad-07eed42da980","Type":"ContainerDied","Data":"5cff7d5aad87d60989b2bf59a0379234a4f1c1981970e772e72e7f21b9116442"}
Oct 30 17:50:55.464028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.463885    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cff7d5aad87d60989b2bf59a0379234a4f1c1981970e772e72e7f21b9116442"
Oct 30 17:50:55.474694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.474676    5534 scope.go:122] "RemoveContainer" containerID="fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"
Oct 30 17:50:55.488484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.488466    5534 scope.go:122] "RemoveContainer" containerID="c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1"
Oct 30 17:50:55.488841 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:55.488812    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1\": container with ID starting with c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1 not found: ID does not exist" containerID="c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1"
Oct 30 17:50:55.488942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.488850    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1"} err="failed to get container status \"c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1\": rpc error: code = NotFound desc = could not find container \"c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1\": container with ID starting with c51a26df64efaf4ce3befc2fbad79a74ae43618faf19829acbb18253bd5869b1 not found: ID does not exist"
Oct 30 17:50:55.488942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.488869    5534 scope.go:122] "RemoveContainer" containerID="fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"
Oct 30 17:50:55.489127 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:50:55.489111    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a\": container with ID starting with fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a not found: ID does not exist" containerID="fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"
Oct 30 17:50:55.489171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.489130    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a"} err="failed to get container status \"fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a\": rpc error: code = NotFound desc = could not find container \"fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a\": container with ID starting with fde46a6fe5abd076ff51f95cac924d28413679457f4fbdb254ccc3d16aaf5e4a not found: ID does not exist"
Oct 30 17:50:55.533232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.533141    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"]
Oct 30 17:50:55.545049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:55.545020    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ff3-openshift-default-58d9fcffb9-rscr4"]
Oct 30 17:50:56.373448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:56.373412    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" path="/var/lib/kubelet/pods/2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10/volumes"
Oct 30 17:50:57.032374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:57.032343    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:57.588513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:57.588476    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"]
Oct 30 17:50:57.729876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:57.729841    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:57.847913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:57.847880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:58.728593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:58.728553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:59.267088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:59.267055    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:50:59.757595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:50:59.757558    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:00.195702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.195666    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"]
Oct 30 17:51:00.196769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.196742    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy"
Oct 30 17:51:00.196769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.196769    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy"
Oct 30 17:51:00.196965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.196833    5534 cpu_manager.go:414] "containerMap: removing container" podUID="102a1711-2402-4fb0-81ad-07eed42da980" containerName="revoke-keys"
Oct 30 17:51:00.196965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.196840    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102a1711-2402-4fb0-81ad-07eed42da980" containerName="revoke-keys"
Oct 30 17:51:00.197095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.197046    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy"
Oct 30 17:51:00.197095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.197071    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="102a1711-2402-4fb0-81ad-07eed42da980" containerName="revoke-keys"
Oct 30 17:51:00.203516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.203493    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.228145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.228111    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"]
Oct 30 17:51:00.233588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.233557    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-01b925-openshift-default-dockercfg-mn67p\""
Oct 30 17:51:00.244865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.244835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:00.249015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.248985    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249019    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249040    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249156    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249192    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249224    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249315    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249432    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.249599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.249584    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxrb\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.362973    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363080    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363166    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363294    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxrb\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363336    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363363    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363390    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363447    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.363668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.363494    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.378686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.365053    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.378686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.376010    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.378686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.376265    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.388856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.388811    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.393623 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.393595    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.434032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.433954    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.481745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.481673    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxrb\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.488774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.482581    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.496697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.496613    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.520215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.520187    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:00.862161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.862118    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:00.876121 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:00.876089    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed38c99_08e0_476e_aa19_6602e5073655.slice/crio-120179bad04bc958f2ecde9d2abc901b40ec5fa989d6488c0112a7621c6bf3f7 WatchSource:0}: Error finding container 120179bad04bc958f2ecde9d2abc901b40ec5fa989d6488c0112a7621c6bf3f7: Status 404 returned error can't find the container with id 120179bad04bc958f2ecde9d2abc901b40ec5fa989d6488c0112a7621c6bf3f7
Oct 30 17:51:00.877136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.877107    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"]
Oct 30 17:51:00.879290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.879248    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:00.879380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.879333    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:00.879380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.879375    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:00.972872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:00.972841    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:01.224850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:01.224764    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:01.533889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:01.533808    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" event={"ID":"4ed38c99-08e0-476e-aa19-6602e5073655","Type":"ContainerStarted","Data":"5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"}
Oct 30 17:51:01.533889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:01.533862    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" event={"ID":"4ed38c99-08e0-476e-aa19-6602e5073655","Type":"ContainerStarted","Data":"120179bad04bc958f2ecde9d2abc901b40ec5fa989d6488c0112a7621c6bf3f7"}
Oct 30 17:51:01.583035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:01.582973    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podStartSLOduration=1.5829574320000002 podStartE2EDuration="1.582957432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:01.578512514 +0000 UTC m=+2267.340556329" watchObservedRunningTime="2026-10-30 17:51:01.582957432 +0000 UTC m=+2267.345001246"
Oct 30 17:51:02.520727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.520685    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:02.527518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.527490    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:02.542578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.542545    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:02.543532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.543512    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:51:02.589167 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.589082    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerName="payload-processing" containerID="cri-o://d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53" gracePeriod=25
Oct 30 17:51:02.606511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.606478    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:02.919301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:02.919265    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:51:04.020680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.020608    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"]
Oct 30 17:51:04.021424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.021408    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy"
Oct 30 17:51:04.021471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.021425    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy"
Oct 30 17:51:04.021562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.021552    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f00bb78-edd8-4a6a-88e6-1e3e0b66ba10" containerName="istio-proxy"
Oct 30 17:51:04.025998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.025966    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.037695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.037670    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:04.046132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.046107    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bcdac0-openshift-default-dockercfg-6hq5t\""
Oct 30 17:51:04.059902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.059866    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"]
Oct 30 17:51:04.122897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.122864    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.122926    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123069    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123199    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9wb\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123233    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123308    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123353    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123386    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.123480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.123467    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.170771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.170726    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:04.225332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225292    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225354    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225390    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225419    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225455    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225501    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225597    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225806    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9wb\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.225884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225849    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.226090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225919    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.226090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.225996    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.226090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.226060    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.226201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.226122    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.226303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.226279    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.228155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.228136    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.228426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.228410    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.252233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.252194    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.252418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.252394    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9wb\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.343599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.343565    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:04.526328 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.526293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:04.687076 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:04.687036    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992a0c91_f81c_4e53_8217_8bb93a9353b8.slice/crio-2b4168cb30ad7002c7403fdadbb65af10cac49dce51237cc15558bb4339d54fd WatchSource:0}: Error finding container 2b4168cb30ad7002c7403fdadbb65af10cac49dce51237cc15558bb4339d54fd: Status 404 returned error can't find the container with id 2b4168cb30ad7002c7403fdadbb65af10cac49dce51237cc15558bb4339d54fd
Oct 30 17:51:04.688657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.688604    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"]
Oct 30 17:51:04.690736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.690695    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:04.690863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.690786    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:04.690863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:04.690827    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:05.610675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:05.609698    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" event={"ID":"992a0c91-f81c-4e53-8217-8bb93a9353b8","Type":"ContainerStarted","Data":"4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d"}
Oct 30 17:51:05.610675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:05.609769    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" event={"ID":"992a0c91-f81c-4e53-8217-8bb93a9353b8","Type":"ContainerStarted","Data":"2b4168cb30ad7002c7403fdadbb65af10cac49dce51237cc15558bb4339d54fd"}
Oct 30 17:51:05.706654 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:05.706560    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podStartSLOduration=2.706537198 podStartE2EDuration="2.706537198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:05.680149524 +0000 UTC m=+2271.442193360" watchObservedRunningTime="2026-10-30 17:51:05.706537198 +0000 UTC m=+2271.468581010"
Oct 30 17:51:05.732929 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:05.732891    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:05.900674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:05.900553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:06.157721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.152299    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"]
Oct 30 17:51:06.182670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.158430    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.276677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.187637    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-01b925-dockercfg-zj4gz\""
Oct 30 17:51:06.276677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.188355    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-01b925\""
Oct 30 17:51:06.276677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.244259    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfgb\" (UniqueName: \"kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb\") pod \"payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.276677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.244304    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.276677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.262997    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"]
Oct 30 17:51:06.343888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.343854    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:06.345618 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.345586    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfgb\" (UniqueName: \"kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb\") pod \"payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.345809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.345627    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.345809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.345678    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 17:51:06.345809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.345717    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 17:51:06.346387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.346369    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.389789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.381708    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfgb\" (UniqueName: \"kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb\") pod \"payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.596784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.596752    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:06.647790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.644470    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:06.991715 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:06.983460    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db1ff6e_79b1_4ef4_83bb_716435f6f1eb.slice/crio-7da2c04ac901defb21f5f5f1319d3904f150ad8c114e19b4dff5c513b9b91b3e WatchSource:0}: Error finding container 7da2c04ac901defb21f5f5f1319d3904f150ad8c114e19b4dff5c513b9b91b3e: Status 404 returned error can't find the container with id 7da2c04ac901defb21f5f5f1319d3904f150ad8c114e19b4dff5c513b9b91b3e
Oct 30 17:51:06.995613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:06.995590    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"]
Oct 30 17:51:07.344694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:07.344662    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 17:51:07.344940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:07.344725    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 17:51:07.629935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:07.629846    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" event={"ID":"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb","Type":"ContainerStarted","Data":"c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326"}
Oct 30 17:51:07.629935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:07.629906    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" event={"ID":"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb","Type":"ContainerStarted","Data":"7da2c04ac901defb21f5f5f1319d3904f150ad8c114e19b4dff5c513b9b91b3e"}
Oct 30 17:51:07.629935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:07.629935    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:07.689355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:07.689288    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" podStartSLOduration=2.689266905 podStartE2EDuration="2.689266905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:07.674889165 +0000 UTC m=+2273.436932979" watchObservedRunningTime="2026-10-30 17:51:07.689266905 +0000 UTC m=+2273.451310723"
Oct 30 17:51:08.344604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:08.344572    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 17:51:08.344813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:08.344626    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 17:51:08.522836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:08.522791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:09.344014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:09.343983    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 17:51:09.344436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:09.344039    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 17:51:10.175670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.173757    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:10.344668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.344390    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 17:51:10.344668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.344454    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 17:51:10.421584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.418278    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"]
Oct 30 17:51:10.425940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.425865    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"]
Oct 30 17:51:10.426244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.426223    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.435792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.434859    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jj8b\" (UniqueName: \"kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.435792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.435118    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.435792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.435138    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.441981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.441955    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:51:10.442114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.442022    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:51:10.535997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.535963    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.535997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.536001    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.536222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.536056    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jj8b\" (UniqueName: \"kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.536695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.536674    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.538580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.538554    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.595336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.595303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jj8b\" (UniqueName: \"kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b\") pod \"maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:10.605385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.605351    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:10.746933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:10.746853    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:11.107583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:11.107548    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"]
Oct 30 17:51:11.108414 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:11.108385    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6544114_1999_4bd4_9e47_9938cfcc6817.slice/crio-0e059358484f6b5a0d409ccebb481a67264203015d568584820841022fa9caa1 WatchSource:0}: Error finding container 0e059358484f6b5a0d409ccebb481a67264203015d568584820841022fa9caa1: Status 404 returned error can't find the container with id 0e059358484f6b5a0d409ccebb481a67264203015d568584820841022fa9caa1
Oct 30 17:51:11.344708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:11.344673    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 17:51:11.345101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:11.344732    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 17:51:11.736888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:11.736779    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn" event={"ID":"e6544114-1999-4bd4-9e47-9938cfcc6817","Type":"ContainerStarted","Data":"96008a69d82648e8fa2b259a16fd4b365ef89b652b3a372a74f649718f00471b"}
Oct 30 17:51:11.736888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:11.736846    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn" event={"ID":"e6544114-1999-4bd4-9e47-9938cfcc6817","Type":"ContainerStarted","Data":"0e059358484f6b5a0d409ccebb481a67264203015d568584820841022fa9caa1"}
Oct 30 17:51:11.782314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:11.780632    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn" podStartSLOduration=1.78061213 podStartE2EDuration="1.78061213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:11.779202201 +0000 UTC m=+2277.541246016" watchObservedRunningTime="2026-10-30 17:51:11.78061213 +0000 UTC m=+2277.542655944"
Oct 30 17:51:12.348953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.348923    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:12.349438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.349425    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:12.350113 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.350091    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:51:12.529170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.529070    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:12.746226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.746189    5534 generic.go:356] "Generic (PLEG): container finished" podID="e6544114-1999-4bd4-9e47-9938cfcc6817" containerID="96008a69d82648e8fa2b259a16fd4b365ef89b652b3a372a74f649718f00471b" exitCode=0
Oct 30 17:51:12.746418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.746263    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn" event={"ID":"e6544114-1999-4bd4-9e47-9938cfcc6817","Type":"ContainerDied","Data":"96008a69d82648e8fa2b259a16fd4b365ef89b652b3a372a74f649718f00471b"}
Oct 30 17:51:12.920563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:12.920523    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:51:13.927089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:13.927056    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:14.011415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.011303    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b\" (UniqueName: \"kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b\") pod \"e6544114-1999-4bd4-9e47-9938cfcc6817\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") "
Oct 30 17:51:14.011615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.011595    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca\") pod \"e6544114-1999-4bd4-9e47-9938cfcc6817\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") "
Oct 30 17:51:14.011736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.011718    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca\") pod \"e6544114-1999-4bd4-9e47-9938cfcc6817\" (UID: \"e6544114-1999-4bd4-9e47-9938cfcc6817\") "
Oct 30 17:51:14.012035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.012007    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca" pod "e6544114-1999-4bd4-9e47-9938cfcc6817" (UID: "e6544114-1999-4bd4-9e47-9938cfcc6817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:14.012703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.012628    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:14.013949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.013916    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b" pod "e6544114-1999-4bd4-9e47-9938cfcc6817" (UID: "e6544114-1999-4bd4-9e47-9938cfcc6817"). InnerVolumeSpecName "kube-api-access-6jj8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:14.014049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.013975    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca" pod "e6544114-1999-4bd4-9e47-9938cfcc6817" (UID: "e6544114-1999-4bd4-9e47-9938cfcc6817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:51:14.114100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.114065    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6544114-1999-4bd4-9e47-9938cfcc6817-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:14.114100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.114096    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jj8b\" (UniqueName: \"kubernetes.io/projected/e6544114-1999-4bd4-9e47-9938cfcc6817-kube-api-access-6jj8b\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:14.316147 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.316041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:14.746833 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.746797    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:14.765314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.765282    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"
Oct 30 17:51:14.765314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.765299    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn" event={"ID":"e6544114-1999-4bd4-9e47-9938cfcc6817","Type":"ContainerDied","Data":"0e059358484f6b5a0d409ccebb481a67264203015d568584820841022fa9caa1"}
Oct 30 17:51:14.765559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:14.765354    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e059358484f6b5a0d409ccebb481a67264203015d568584820841022fa9caa1"
Oct 30 17:51:15.249076 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:15.249039    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:51:16.205343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:16.205303    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:16.973835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:16.973798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:18.209391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:18.202672    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:18.231356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:18.231322    5534 scope.go:122] "RemoveContainer" containerID="a707a57c47253203151a6e9bbb70ee62a87cb1dc4528dea6cb9643cdfdb54099"
Oct 30 17:51:18.685696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:18.680889    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 17:51:20.241949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:20.241918    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:20.935606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:20.935569    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"]
Oct 30 17:51:20.956226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:20.956191    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"]
Oct 30 17:51:21.067581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:21.067550    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:21.195466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:21.195387    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:22.248941 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:22.248908    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:22.889864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:22.889835    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:51:23.016694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.016666    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc\" (UniqueName: \"kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc\") pod \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") "
Oct 30 17:51:23.016860 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.016761    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume\") pod \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\" (UID: \"64b8ffb3-e17f-434f-8a2d-c1caf4894554\") "
Oct 30 17:51:23.017158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.017129    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume" pod "64b8ffb3-e17f-434f-8a2d-c1caf4894554" (UID: "64b8ffb3-e17f-434f-8a2d-c1caf4894554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:23.018998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.018971    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc" pod "64b8ffb3-e17f-434f-8a2d-c1caf4894554" (UID: "64b8ffb3-e17f-434f-8a2d-c1caf4894554"). InnerVolumeSpecName "kube-api-access-jjbcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:23.118599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.118504    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjbcc\" (UniqueName: \"kubernetes.io/projected/64b8ffb3-e17f-434f-8a2d-c1caf4894554-kube-api-access-jjbcc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:23.118599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.118530    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b8ffb3-e17f-434f-8a2d-c1caf4894554-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:23.119608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.119583    5534 generic.go:356] "Generic (PLEG): container finished" podID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerID="d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53" exitCode=0
Oct 30 17:51:23.119715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.119617    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" event={"ID":"64b8ffb3-e17f-434f-8a2d-c1caf4894554","Type":"ContainerDied","Data":"d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53"}
Oct 30 17:51:23.119715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.119628    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"
Oct 30 17:51:23.119715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.119671    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f" event={"ID":"64b8ffb3-e17f-434f-8a2d-c1caf4894554","Type":"ContainerDied","Data":"40da5446e7f98b8ca50cd7b0e4e378c522d3d553dec96fe93d9c994f534c488f"}
Oct 30 17:51:23.119715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.119693    5534 scope.go:122] "RemoveContainer" containerID="d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53"
Oct 30 17:51:23.131518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.131500    5534 scope.go:122] "RemoveContainer" containerID="d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53"
Oct 30 17:51:23.131820 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:23.131801    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53\": container with ID starting with d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53 not found: ID does not exist" containerID="d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53"
Oct 30 17:51:23.131885 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.131827    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53"} err="failed to get container status \"d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53\": rpc error: code = NotFound desc = could not find container \"d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53\": container with ID starting with d145116183d9579b38d0126e3f660c215166bcc29b9cb6a661661ad67f7cca53 not found: ID does not exist"
Oct 30 17:51:23.173428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.173396    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"]
Oct 30 17:51:23.180399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.180279    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fb0d-64cf4c6bcd-6b75f"]
Oct 30 17:51:23.857179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:23.857140    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:24.382502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:24.382471    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" path="/var/lib/kubelet/pods/64b8ffb3-e17f-434f-8a2d-c1caf4894554/volumes"
Oct 30 17:51:25.021346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.021300    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:25.529534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.529503    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"]
Oct 30 17:51:25.530528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.530508    5534 cpu_manager.go:414] "containerMap: removing container" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerName="payload-processing"
Oct 30 17:51:25.530589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.530533    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerName="payload-processing"
Oct 30 17:51:25.530589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.530545    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e6544114-1999-4bd4-9e47-9938cfcc6817" containerName="revoke-keys"
Oct 30 17:51:25.530589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.530552    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6544114-1999-4bd4-9e47-9938cfcc6817" containerName="revoke-keys"
Oct 30 17:51:25.530739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.530727    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b8ffb3-e17f-434f-8a2d-c1caf4894554" containerName="payload-processing"
Oct 30 17:51:25.530784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.530744    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6544114-1999-4bd4-9e47-9938cfcc6817" containerName="revoke-keys"
Oct 30 17:51:25.535918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.535898    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.551777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.551752    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:51:25.559261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.559227    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"]
Oct 30 17:51:25.563894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.563852    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:51:25.639169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.639140    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:25.654839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.654808    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.655010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.654861    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.655111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.655090    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7r6k\" (UniqueName: \"kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.757209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.757161    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.757427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.757224    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.757427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.757385    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7r6k\" (UniqueName: \"kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.760267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.758887    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.775890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.775851    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.782331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.782243    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7r6k\" (UniqueName: \"kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k\") pod \"maas-api-revoke-sub-65035508-xzxt4\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.852699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.852660    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:25.937826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.936750    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" containerName="payload-processing" containerID="cri-o://2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95" gracePeriod=25
Oct 30 17:51:25.957433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:25.956811    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerName="payload-pre-processing" containerID="cri-o://24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08" gracePeriod=25
Oct 30 17:51:26.172417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:26.171144    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"]
Oct 30 17:51:26.176827 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:26.176782    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181ba2cf_0774_4ab2_abec_60d6bbdf65a9.slice/crio-f97616ee9459e082ac7584c43da719dd27a9c4f8595c222786948d61a71195fb WatchSource:0}: Error finding container f97616ee9459e082ac7584c43da719dd27a9c4f8595c222786948d61a71195fb: Status 404 returned error can't find the container with id f97616ee9459e082ac7584c43da719dd27a9c4f8595c222786948d61a71195fb
Oct 30 17:51:27.197046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.197002    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4" event={"ID":"181ba2cf-0774-4ab2-abec-60d6bbdf65a9","Type":"ContainerStarted","Data":"445104d711b45bf74219f5728808bf81189d89bfc152e2cf2f81d6ca9e765ff2"}
Oct 30 17:51:27.197539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.197069    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4" event={"ID":"181ba2cf-0774-4ab2-abec-60d6bbdf65a9","Type":"ContainerStarted","Data":"f97616ee9459e082ac7584c43da719dd27a9c4f8595c222786948d61a71195fb"}
Oct 30 17:51:27.241059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.241004    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4" podStartSLOduration=2.24098807 podStartE2EDuration="2.24098807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:27.23736235 +0000 UTC m=+2292.999406163" watchObservedRunningTime="2026-10-30 17:51:27.24098807 +0000 UTC m=+2293.003031882"
Oct 30 17:51:27.274776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.274741    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"]
Oct 30 17:51:27.275014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.274986    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy" containerID="cri-o://703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be" gracePeriod=30
Oct 30 17:51:27.426982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.426947    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:27.578036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:27.577944    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:28.210418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:28.210380    5534 generic.go:356] "Generic (PLEG): container finished" podID="181ba2cf-0774-4ab2-abec-60d6bbdf65a9" containerID="445104d711b45bf74219f5728808bf81189d89bfc152e2cf2f81d6ca9e765ff2" exitCode=0
Oct 30 17:51:28.210903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:28.210467    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4" event={"ID":"181ba2cf-0774-4ab2-abec-60d6bbdf65a9","Type":"ContainerDied","Data":"445104d711b45bf74219f5728808bf81189d89bfc152e2cf2f81d6ca9e765ff2"}
Oct 30 17:51:28.864907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:28.864876    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:29.502732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.502331    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:29.550092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.550027    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:29.641893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.641843    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:29.662069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.662031    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca\") pod \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") "
Oct 30 17:51:29.662284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.662152    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca\") pod \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") "
Oct 30 17:51:29.662380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.662355    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k\" (UniqueName: \"kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k\") pod \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\" (UID: \"181ba2cf-0774-4ab2-abec-60d6bbdf65a9\") "
Oct 30 17:51:29.669872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.669836    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca" pod "181ba2cf-0774-4ab2-abec-60d6bbdf65a9" (UID: "181ba2cf-0774-4ab2-abec-60d6bbdf65a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:29.681756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.681726    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"]
Oct 30 17:51:29.682179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.682153    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k" pod "181ba2cf-0774-4ab2-abec-60d6bbdf65a9" (UID: "181ba2cf-0774-4ab2-abec-60d6bbdf65a9"). InnerVolumeSpecName "kube-api-access-b7r6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:29.692818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.691853    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca" pod "181ba2cf-0774-4ab2-abec-60d6bbdf65a9" (UID: "181ba2cf-0774-4ab2-abec-60d6bbdf65a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:51:29.692818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.692325    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fb0d-68w5f"]
Oct 30 17:51:29.766502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.766460    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:29.766502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.766492    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:29.766502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.766508    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7r6k\" (UniqueName: \"kubernetes.io/projected/181ba2cf-0774-4ab2-abec-60d6bbdf65a9-kube-api-access-b7r6k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:29.904736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:29.904637    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:30.101632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.101588    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:51:30.101894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.101868    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:51:30.167411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.167319    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:30.279449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.279409    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4" event={"ID":"181ba2cf-0774-4ab2-abec-60d6bbdf65a9","Type":"ContainerDied","Data":"f97616ee9459e082ac7584c43da719dd27a9c4f8595c222786948d61a71195fb"}
Oct 30 17:51:30.279707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.279471    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f97616ee9459e082ac7584c43da719dd27a9c4f8595c222786948d61a71195fb"
Oct 30 17:51:30.279707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.279551    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"
Oct 30 17:51:30.373513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:30.373479    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86ae4bd-e84f-4f91-b262-160ba2b36c66" path="/var/lib/kubelet/pods/b86ae4bd-e84f-4f91-b262-160ba2b36c66/volumes"
Oct 30 17:51:31.729564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:31.729523    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:32.581989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.581965    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:51:32.711336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711284    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711478    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711597    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711693    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711784 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711753    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711794    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711841    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.711901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711870    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.712019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711909    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq\") pod \"85be8266-684d-42b7-9940-a9d096dedaa4\" (UID: \"85be8266-684d-42b7-9940-a9d096dedaa4\") "
Oct 30 17:51:32.712019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.711912    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:32.712124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712038    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:32.712238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712117    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:32.712425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712294    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:32.712425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712304    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:32.712539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712501    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.712539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712515    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85be8266-684d-42b7-9940-a9d096dedaa4-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.712539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712524    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.712539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712531    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.712790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.712543    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.714118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.714083    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:32.714240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.714144    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:51:32.714381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.714367    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:32.714506 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.714485    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq" pod "85be8266-684d-42b7-9940-a9d096dedaa4" (UID: "85be8266-684d-42b7-9940-a9d096dedaa4"). InnerVolumeSpecName "kube-api-access-xtbrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:32.814200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.814118    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85be8266-684d-42b7-9940-a9d096dedaa4-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.814200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.814147    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85be8266-684d-42b7-9940-a9d096dedaa4-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.814200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.814165    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:32.814200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:32.814175    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtbrq\" (UniqueName: \"kubernetes.io/projected/85be8266-684d-42b7-9940-a9d096dedaa4-kube-api-access-xtbrq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:33.308813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.308783    5534 generic.go:356] "Generic (PLEG): container finished" podID="85be8266-684d-42b7-9940-a9d096dedaa4" containerID="703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be" exitCode=0
Oct 30 17:51:33.308964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.308819    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" event={"ID":"85be8266-684d-42b7-9940-a9d096dedaa4","Type":"ContainerDied","Data":"703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be"}
Oct 30 17:51:33.308964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.308834    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"
Oct 30 17:51:33.308964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.308860    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x" event={"ID":"85be8266-684d-42b7-9940-a9d096dedaa4","Type":"ContainerDied","Data":"005f8cacbb5df4af6ddfdfd7f6dbe6be10d03fb0439512608a3f12ff928ea00e"}
Oct 30 17:51:33.308964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.308881    5534 scope.go:122] "RemoveContainer" containerID="703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be"
Oct 30 17:51:33.321972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.321884    5534 scope.go:122] "RemoveContainer" containerID="703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be"
Oct 30 17:51:33.322212 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:33.322189    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be\": container with ID starting with 703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be not found: ID does not exist" containerID="703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be"
Oct 30 17:51:33.322298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.322221    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be"} err="failed to get container status \"703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be\": rpc error: code = NotFound desc = could not find container \"703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be\": container with ID starting with 703a4e7daeac19f0008958f477f3578c3ffedbe0d230859d09fb1f0da78872be not found: ID does not exist"
Oct 30 17:51:33.371149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.371089    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:33.386917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.386872    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"]
Oct 30 17:51:33.393470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:33.393438    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-xdz6x"]
Oct 30 17:51:34.371780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:34.371751    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" path="/var/lib/kubelet/pods/85be8266-684d-42b7-9940-a9d096dedaa4/volumes"
Oct 30 17:51:35.356231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:35.356196    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:37.152018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:37.151984    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:37.862464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:37.862423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:37.994610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:37.994571    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:38.236312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:38.236225    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:39.141257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.141225    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:39.323525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.323491    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:39.600080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.600041    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"]
Oct 30 17:51:39.601165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.601141    5534 cpu_manager.go:414] "containerMap: removing container" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy"
Oct 30 17:51:39.601165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.601164    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy"
Oct 30 17:51:39.601397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.601181    5534 cpu_manager.go:414] "containerMap: removing container" podUID="181ba2cf-0774-4ab2-abec-60d6bbdf65a9" containerName="revoke-keys"
Oct 30 17:51:39.601397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.601189    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181ba2cf-0774-4ab2-abec-60d6bbdf65a9" containerName="revoke-keys"
Oct 30 17:51:39.601523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.601432    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="85be8266-684d-42b7-9940-a9d096dedaa4" containerName="istio-proxy"
Oct 30 17:51:39.601523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.601448    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="181ba2cf-0774-4ab2-abec-60d6bbdf65a9" containerName="revoke-keys"
Oct 30 17:51:39.605413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.605387    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.620504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.620202    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a2a6cb-openshift-default-dockercfg-qs2lw\""
Oct 30 17:51:39.651035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.651002    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"]
Oct 30 17:51:39.665009 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.664964    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:39.776669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.758714    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.776974    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.777122    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.777235    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.777416    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.777582    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.777783    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.777914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.777902    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.778033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.778021    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlks\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.880606    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.880744    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.880892    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.881019    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.881065    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.881118    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlks\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.881299    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.881336    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.884716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.881377    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.891339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.890902    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.892202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.892173    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.896663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.892870    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.896846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.895091    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.900721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.895427    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.900872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.896022    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.901052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.896972    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.933688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.933604    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.933688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.933605    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlks\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:39.958245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:39.958216    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:40.101451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.101343    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:51:40.101957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.101926    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:51:40.120101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.120068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:40.288372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.288339    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"]
Oct 30 17:51:40.288717 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:40.288695    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd9f802_7d3b_45e6_9e0f_447ae8a11fa9.slice/crio-b26625f6d95c3dbe8b849daede36ec1da295f3c2fc835523ca240de62c60067b WatchSource:0}: Error finding container b26625f6d95c3dbe8b849daede36ec1da295f3c2fc835523ca240de62c60067b: Status 404 returned error can't find the container with id b26625f6d95c3dbe8b849daede36ec1da295f3c2fc835523ca240de62c60067b
Oct 30 17:51:40.291573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.291543    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:40.291673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.291609    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:40.291673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.291668    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:51:40.499854 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.499816    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" event={"ID":"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9","Type":"ContainerStarted","Data":"ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"}
Oct 30 17:51:40.500292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.499885    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" event={"ID":"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9","Type":"ContainerStarted","Data":"b26625f6d95c3dbe8b849daede36ec1da295f3c2fc835523ca240de62c60067b"}
Oct 30 17:51:40.550055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.549929    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podStartSLOduration=1.54991374 podStartE2EDuration="1.54991374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:40.544449339 +0000 UTC m=+2306.306493162" watchObservedRunningTime="2026-10-30 17:51:40.54991374 +0000 UTC m=+2306.311957552"
Oct 30 17:51:40.959225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:40.959191    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:41.098712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:41.098680    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:41.966108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:41.966074    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:42.518425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:42.518383    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:42.519558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:42.519536    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:51:42.629916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:42.629880    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"]
Oct 30 17:51:42.630396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:42.630367    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api" containerID="cri-o://99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96" gracePeriod=30
Oct 30 17:51:42.704402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:42.704362    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:43.023494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.023009    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:43.033229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.033194    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:51:43.035072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.035052    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls\") pod \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") "
Oct 30 17:51:43.035305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.035284    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8\" (UniqueName: \"kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8\") pod \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") "
Oct 30 17:51:43.035367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.035326    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs\") pod \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") "
Oct 30 17:51:43.035367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.035359    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca\") pod \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\" (UID: \"7ac41727-4af1-48f9-b41d-2b7186f34a6d\") "
Oct 30 17:51:43.035971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.035946    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca" pod "7ac41727-4af1-48f9-b41d-2b7186f34a6d" (UID: "7ac41727-4af1-48f9-b41d-2b7186f34a6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:43.036207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.036189    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41727-4af1-48f9-b41d-2b7186f34a6d-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:43.037612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.037584    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls" pod "7ac41727-4af1-48f9-b41d-2b7186f34a6d" (UID: "7ac41727-4af1-48f9-b41d-2b7186f34a6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:51:43.038077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.038056    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8" pod "7ac41727-4af1-48f9-b41d-2b7186f34a6d" (UID: "7ac41727-4af1-48f9-b41d-2b7186f34a6d"). InnerVolumeSpecName "kube-api-access-5vcf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:43.038176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.038158    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs" pod "7ac41727-4af1-48f9-b41d-2b7186f34a6d" (UID: "7ac41727-4af1-48f9-b41d-2b7186f34a6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:51:43.138542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.138314    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vcf8\" (UniqueName: \"kubernetes.io/projected/7ac41727-4af1-48f9-b41d-2b7186f34a6d-kube-api-access-5vcf8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:43.138813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.138555    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:43.138813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.138571    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ac41727-4af1-48f9-b41d-2b7186f34a6d-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:43.207708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.207571    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:43.376222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.376189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:43.472832 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.472744    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"]
Oct 30 17:51:43.479295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.479268    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38f445-lxfsn"]
Oct 30 17:51:43.509971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.509934    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:43.528208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.528171    5534 generic.go:356] "Generic (PLEG): container finished" podID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerID="99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96" exitCode=0
Oct 30 17:51:43.528375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.528282    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"
Oct 30 17:51:43.528375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.528295    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" event={"ID":"7ac41727-4af1-48f9-b41d-2b7186f34a6d","Type":"ContainerDied","Data":"99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96"}
Oct 30 17:51:43.528375 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.528353    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" event={"ID":"7ac41727-4af1-48f9-b41d-2b7186f34a6d","Type":"ContainerDied","Data":"9e12ece60bf060bac0d712403f9a6dac5183c834d37a3d1b8d52afe9a76941ba"}
Oct 30 17:51:43.528484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.528383    5534 scope.go:122] "RemoveContainer" containerID="99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96"
Oct 30 17:51:43.543604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.543585    5534 scope.go:122] "RemoveContainer" containerID="b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"
Oct 30 17:51:43.557364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.557347    5534 scope.go:122] "RemoveContainer" containerID="99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96"
Oct 30 17:51:43.557740 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:43.557706    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96\": container with ID starting with 99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96 not found: ID does not exist" containerID="99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96"
Oct 30 17:51:43.557835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.557753    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96"} err="failed to get container status \"99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96\": rpc error: code = NotFound desc = could not find container \"99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96\": container with ID starting with 99f449cc6e1f6dc9d47db48a2962d9e7ee7cfe50594cbc52f642297dafe80f96 not found: ID does not exist"
Oct 30 17:51:43.557835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.557779    5534 scope.go:122] "RemoveContainer" containerID="b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"
Oct 30 17:51:43.558127 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:43.558086    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1\": container with ID starting with b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1 not found: ID does not exist" containerID="b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"
Oct 30 17:51:43.558263 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.558125    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1"} err="failed to get container status \"b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1\": rpc error: code = NotFound desc = could not find container \"b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1\": container with ID starting with b87facb151cab22a2bbcf8381a5e6de4edaa4436d44930f79d8b6a01a74c91d1 not found: ID does not exist"
Oct 30 17:51:43.594749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.594705    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"]
Oct 30 17:51:43.597061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.597010    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w"]
Oct 30 17:51:43.974602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:43.974567    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:44.373332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:44.373302    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" path="/var/lib/kubelet/pods/7ac41727-4af1-48f9-b41d-2b7186f34a6d/volumes"
Oct 30 17:51:44.373812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:44.373797    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6544114-1999-4bd4-9e47-9938cfcc6817" path="/var/lib/kubelet/pods/e6544114-1999-4bd4-9e47-9938cfcc6817/volumes"
Oct 30 17:51:44.864973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:44.864941    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:44.976212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:44.976180    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:45.031531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.031492    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"]
Oct 30 17:51:45.032277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.032262    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api"
Oct 30 17:51:45.032360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.032279    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api"
Oct 30 17:51:45.032428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.032419    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api"
Oct 30 17:51:45.032470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.032434    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api"
Oct 30 17:51:45.037764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.037743    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.059223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.059196    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"]
Oct 30 17:51:45.069796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.065373    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a2a6cb\""
Oct 30 17:51:45.069796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.065513    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a2a6cb-dockercfg-sjp6w\""
Oct 30 17:51:45.167720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.160439    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.167720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.160714    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsdb\" (UniqueName: \"kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb\") pod \"payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.306617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.306572    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsdb\" (UniqueName: \"kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb\") pod \"payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.306888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.306792    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.308214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.307889    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.333466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.333436    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"]
Oct 30 17:51:45.337152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.334933    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api"
Oct 30 17:51:45.337393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.337373    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api"
Oct 30 17:51:45.353700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.346527    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsdb\" (UniqueName: \"kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb\") pod \"payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.353700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.348355    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.353700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.352708    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"]
Oct 30 17:51:45.354731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.354712    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:45.388602 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:45.388487    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:51:45.509470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.509430    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gr2b\" (UniqueName: \"kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b\") pod \"payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.510225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.510198    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.611561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.611527    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.611782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.611594    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gr2b\" (UniqueName: \"kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b\") pod \"payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.612264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.612239    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.636848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.636823    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gr2b\" (UniqueName: \"kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b\") pod \"payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.655151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.655113    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:45.699416 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.699375    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"]
Oct 30 17:51:45.732875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.732005    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:45.954293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:45.954256    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38f445-67957bcd4c-tqw5w" podUID="7ac41727-4af1-48f9-b41d-2b7186f34a6d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.181:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:51:46.053285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.053251    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"]
Oct 30 17:51:46.057014 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:46.056989    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045cf88a_234e_43fa_936d_d139c5786225.slice/crio-d98e5a2ecdf3cf1410ddc74482f6ffea9b827131175ff8f604e6b723cb7f0c1c WatchSource:0}: Error finding container d98e5a2ecdf3cf1410ddc74482f6ffea9b827131175ff8f604e6b723cb7f0c1c: Status 404 returned error can't find the container with id d98e5a2ecdf3cf1410ddc74482f6ffea9b827131175ff8f604e6b723cb7f0c1c
Oct 30 17:51:46.236879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.236856    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.323063    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.325437    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume\") pod \"8e12ba46-6722-4e2f-9369-895b292d605c\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") "
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.325486    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l\" (UniqueName: \"kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l\") pod \"4cbd05e7-3658-4053-9901-0b1383bfc596\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") "
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.325515    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx\" (UniqueName: \"kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx\") pod \"8e12ba46-6722-4e2f-9369-895b292d605c\" (UID: \"8e12ba46-6722-4e2f-9369-895b292d605c\") "
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.325685    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume\") pod \"4cbd05e7-3658-4053-9901-0b1383bfc596\" (UID: \"4cbd05e7-3658-4053-9901-0b1383bfc596\") "
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.325897    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume" pod "8e12ba46-6722-4e2f-9369-895b292d605c" (UID: "8e12ba46-6722-4e2f-9369-895b292d605c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.326384    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12ba46-6722-4e2f-9369-895b292d605c-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:46.332868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.326761    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume" pod "4cbd05e7-3658-4053-9901-0b1383bfc596" (UID: "4cbd05e7-3658-4053-9901-0b1383bfc596"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:46.333943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.333911    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx" pod "8e12ba46-6722-4e2f-9369-895b292d605c" (UID: "8e12ba46-6722-4e2f-9369-895b292d605c"). InnerVolumeSpecName "kube-api-access-s9khx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:46.334214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.334190    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l" pod "4cbd05e7-3658-4053-9901-0b1383bfc596" (UID: "4cbd05e7-3658-4053-9901-0b1383bfc596"). InnerVolumeSpecName "kube-api-access-k6w7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:46.428190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.428159    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w7l\" (UniqueName: \"kubernetes.io/projected/4cbd05e7-3658-4053-9901-0b1383bfc596-kube-api-access-k6w7l\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:46.428559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.428198    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9khx\" (UniqueName: \"kubernetes.io/projected/8e12ba46-6722-4e2f-9369-895b292d605c-kube-api-access-s9khx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:46.428559 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.428214    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbd05e7-3658-4053-9901-0b1383bfc596-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:46.559557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.559454    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" event={"ID":"045cf88a-234e-43fa-936d-d139c5786225","Type":"ContainerStarted","Data":"2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d"}
Oct 30 17:51:46.559796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.559561    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" event={"ID":"045cf88a-234e-43fa-936d-d139c5786225","Type":"ContainerStarted","Data":"d98e5a2ecdf3cf1410ddc74482f6ffea9b827131175ff8f604e6b723cb7f0c1c"}
Oct 30 17:51:46.559796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.559708    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:46.560903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.560878    5534 generic.go:356] "Generic (PLEG): container finished" podID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerID="24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08" exitCode=0
Oct 30 17:51:46.561047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.560929    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"
Oct 30 17:51:46.561047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.560951    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" event={"ID":"4cbd05e7-3658-4053-9901-0b1383bfc596","Type":"ContainerDied","Data":"24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08"}
Oct 30 17:51:46.561047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.561016    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74" event={"ID":"4cbd05e7-3658-4053-9901-0b1383bfc596","Type":"ContainerDied","Data":"e122fd15d40c5c1bc063d4421fac049617111f6acae72c5ce6d51af22081d51d"}
Oct 30 17:51:46.561195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.561045    5534 scope.go:122] "RemoveContainer" containerID="24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08"
Oct 30 17:51:46.562902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.562874    5534 generic.go:356] "Generic (PLEG): container finished" podID="8e12ba46-6722-4e2f-9369-895b292d605c" containerID="2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95" exitCode=0
Oct 30 17:51:46.563032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.562937    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"
Oct 30 17:51:46.563032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.562985    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" event={"ID":"8e12ba46-6722-4e2f-9369-895b292d605c","Type":"ContainerDied","Data":"2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95"}
Oct 30 17:51:46.563032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.563026    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc" event={"ID":"8e12ba46-6722-4e2f-9369-895b292d605c","Type":"ContainerDied","Data":"5be90bd95875526d6be029430baf91f50890f76c950aabbb93d0ec5eaf0064ef"}
Oct 30 17:51:46.565525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.565313    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" event={"ID":"d6c0b287-2026-4261-a085-073736c390c8","Type":"ContainerStarted","Data":"7ecc029fa2570a64f5ad64d2679d04fb3618dd6c91aabc0327a87ad661ce781d"}
Oct 30 17:51:46.565525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.565361    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" event={"ID":"d6c0b287-2026-4261-a085-073736c390c8","Type":"ContainerStarted","Data":"72870bdf3c5a8871307f7e47c5760bad7d33ea905a90838327751ff111b6ef62"}
Oct 30 17:51:46.565525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.565390    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:46.576916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.576900    5534 scope.go:122] "RemoveContainer" containerID="24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08"
Oct 30 17:51:46.577206 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:46.577182    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08\": container with ID starting with 24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08 not found: ID does not exist" containerID="24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08"
Oct 30 17:51:46.577275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.577221    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08"} err="failed to get container status \"24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08\": rpc error: code = NotFound desc = could not find container \"24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08\": container with ID starting with 24d87643c933e81bc2c32350d6c367d982fe8ca31a07079d86e398077886cc08 not found: ID does not exist"
Oct 30 17:51:46.577275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.577247    5534 scope.go:122] "RemoveContainer" containerID="2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95"
Oct 30 17:51:46.603436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.603386    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" podStartSLOduration=1.603372245 podStartE2EDuration="1.603372245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:46.600357998 +0000 UTC m=+2312.362401814" watchObservedRunningTime="2026-10-30 17:51:46.603372245 +0000 UTC m=+2312.365416057"
Oct 30 17:51:46.637835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.633869    5534 scope.go:122] "RemoveContainer" containerID="2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95"
Oct 30 17:51:46.637835 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:46.634935    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95\": container with ID starting with 2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95 not found: ID does not exist" containerID="2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95"
Oct 30 17:51:46.637835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.634962    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95"} err="failed to get container status \"2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95\": rpc error: code = NotFound desc = could not find container \"2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95\": container with ID starting with 2e3e6d513875d49bb82a985aa1ee045eb711a21427ef0ed979e3cf37c9ab8d95 not found: ID does not exist"
Oct 30 17:51:46.642013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.641981    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"]
Oct 30 17:51:46.649258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.649224    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38f445-549b84c8c5-95z74"]
Oct 30 17:51:46.685450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.685393    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" podStartSLOduration=1.6853780779999998 podStartE2EDuration="1.685378078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:46.68159877 +0000 UTC m=+2312.443642583" watchObservedRunningTime="2026-10-30 17:51:46.685378078 +0000 UTC m=+2312.447421891"
Oct 30 17:51:46.722374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.722343    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"]
Oct 30 17:51:46.726600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:46.726566    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38f445-866f69ddd-zmqcc"]
Oct 30 17:51:47.447794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.447758    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"]
Oct 30 17:51:47.448528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.448510    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerName="payload-pre-processing"
Oct 30 17:51:47.448528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.448528    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerName="payload-pre-processing"
Oct 30 17:51:47.448681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.448548    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" containerName="payload-processing"
Oct 30 17:51:47.448681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.448554    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" containerName="payload-processing"
Oct 30 17:51:47.448681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.448657    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" containerName="payload-pre-processing"
Oct 30 17:51:47.448681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.448668    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" containerName="payload-processing"
Oct 30 17:51:47.453653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.453617    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.477751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.477727    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-01b925-serving-cert\""
Oct 30 17:51:47.485666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.483027    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"]
Oct 30 17:51:47.535514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.535472    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:47.545238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.545210    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.545385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.545261    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp4x\" (UniqueName: \"kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.545467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.545455    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.545529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.545510    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.648788 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.648752    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.650069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.650045    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp4x\" (UniqueName: \"kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.678428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.650251    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.689853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.683125    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.689853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.683205    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.689853 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:47.684505    5534 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-serving-cert: secret "maas-api-e2e-shared-b-w1-01b925-serving-cert" not found
Oct 30 17:51:47.689853 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:47.684584    5534 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls podName:cee5f338-8334-4dde-a55d-dfa15e1ea7e2 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:48.184564839 +0000 UTC m=+2313.946608643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls") pod "maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" (UID: "cee5f338-8334-4dde-a55d-dfa15e1ea7e2") : secret "maas-api-e2e-shared-b-w1-01b925-serving-cert" not found
Oct 30 17:51:47.689853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.688267    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:47.689853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.689039    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:47.699733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:47.695253    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp4x\" (UniqueName: \"kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:48.012687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.012574    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:48.191937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.191906    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:48.194544 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.194517    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:48.370220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.370186    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:48.372304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.372275    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbd05e7-3658-4053-9901-0b1383bfc596" path="/var/lib/kubelet/pods/4cbd05e7-3658-4053-9901-0b1383bfc596/volumes"
Oct 30 17:51:48.372844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.372829    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e12ba46-6722-4e2f-9369-895b292d605c" path="/var/lib/kubelet/pods/8e12ba46-6722-4e2f-9369-895b292d605c/volumes"
Oct 30 17:51:48.701740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:48.699005    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"]
Oct 30 17:51:48.790029 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:48.788740    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee5f338_8334_4dde_a55d_dfa15e1ea7e2.slice/crio-b358a86f0b50c8d21fe18079ffe6a4592f226f4a3d14354d5e2d7bf39cac178b WatchSource:0}: Error finding container b358a86f0b50c8d21fe18079ffe6a4592f226f4a3d14354d5e2d7bf39cac178b: Status 404 returned error can't find the container with id b358a86f0b50c8d21fe18079ffe6a4592f226f4a3d14354d5e2d7bf39cac178b
Oct 30 17:51:49.445955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:49.445867    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:49.608739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:49.608682    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" event={"ID":"cee5f338-8334-4dde-a55d-dfa15e1ea7e2","Type":"ContainerStarted","Data":"0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741"}
Oct 30 17:51:49.609005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:49.608762    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" event={"ID":"cee5f338-8334-4dde-a55d-dfa15e1ea7e2","Type":"ContainerStarted","Data":"b358a86f0b50c8d21fe18079ffe6a4592f226f4a3d14354d5e2d7bf39cac178b"}
Oct 30 17:51:49.609005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:49.608804    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:49.679701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:49.679604    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" podStartSLOduration=2.278170045 podStartE2EDuration="2.679589544s" totalImagesPullingTime="401.419499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:49.678118044 +0000 UTC m=+2315.440161857" watchObservedRunningTime="2026-10-30 17:51:49.679589544 +0000 UTC m=+2315.441633382"
Oct 30 17:51:50.852462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:50.852431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:50.964736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:50.964698    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:51.155717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.155607    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"]
Oct 30 17:51:51.155891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.155855    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy" containerID="cri-o://4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f" gracePeriod=30
Oct 30 17:51:51.185413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.185377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:51.323677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.316302    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:51.735296 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.735267    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"]
Oct 30 17:51:51.740232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.740214    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.765349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.765311    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"]
Oct 30 17:51:51.774084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.773959    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:51:51.774084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.774062    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:51:51.874977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.874934    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.875382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.875087    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9zq\" (UniqueName: \"kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.875382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.875161    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.976582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.976548    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9zq\" (UniqueName: \"kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.976793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.976592    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.976852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.976823    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.977470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.977442    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:51.979367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:51.979343    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:52.001101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:52.001030    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9zq\" (UniqueName: \"kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq\") pod \"maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:52.054755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:52.054722    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:52.381081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:52.381049    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"]
Oct 30 17:51:52.381439 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:51:52.381416    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc526d837_3a5a_484b_9e2b_996d269bd64d.slice/crio-20efb360899e6460310121e7ebb074bec7e5ee22bba0c9409e1ec41369ed2a45 WatchSource:0}: Error finding container 20efb360899e6460310121e7ebb074bec7e5ee22bba0c9409e1ec41369ed2a45: Status 404 returned error can't find the container with id 20efb360899e6460310121e7ebb074bec7e5ee22bba0c9409e1ec41369ed2a45
Oct 30 17:51:52.639242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:52.639156    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8" event={"ID":"c526d837-3a5a-484b-9e2b-996d269bd64d","Type":"ContainerStarted","Data":"20efb360899e6460310121e7ebb074bec7e5ee22bba0c9409e1ec41369ed2a45"}
Oct 30 17:51:53.276508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:53.276472    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:53.650716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:53.650682    5534 generic.go:356] "Generic (PLEG): container finished" podID="c526d837-3a5a-484b-9e2b-996d269bd64d" containerID="374ba5d2f347e018b4ced05b6c97801dcaad9c234f463818b063e3669b96dab4" exitCode=0
Oct 30 17:51:53.650915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:53.650753    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8" event={"ID":"c526d837-3a5a-484b-9e2b-996d269bd64d","Type":"ContainerDied","Data":"374ba5d2f347e018b4ced05b6c97801dcaad9c234f463818b063e3669b96dab4"}
Oct 30 17:51:54.854177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:54.854142    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:55.025817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.025739    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca\") pod \"c526d837-3a5a-484b-9e2b-996d269bd64d\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") "
Oct 30 17:51:55.025977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.025851    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca\") pod \"c526d837-3a5a-484b-9e2b-996d269bd64d\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") "
Oct 30 17:51:55.026056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.026042    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq\" (UniqueName: \"kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq\") pod \"c526d837-3a5a-484b-9e2b-996d269bd64d\" (UID: \"c526d837-3a5a-484b-9e2b-996d269bd64d\") "
Oct 30 17:51:55.026225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.026198    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca" pod "c526d837-3a5a-484b-9e2b-996d269bd64d" (UID: "c526d837-3a5a-484b-9e2b-996d269bd64d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:55.026631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.026602    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:55.028073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.028048    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca" pod "c526d837-3a5a-484b-9e2b-996d269bd64d" (UID: "c526d837-3a5a-484b-9e2b-996d269bd64d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:51:55.028198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.028181    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq" pod "c526d837-3a5a-484b-9e2b-996d269bd64d" (UID: "c526d837-3a5a-484b-9e2b-996d269bd64d"). InnerVolumeSpecName "kube-api-access-rw9zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:55.127984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.127946    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw9zq\" (UniqueName: \"kubernetes.io/projected/c526d837-3a5a-484b-9e2b-996d269bd64d-kube-api-access-rw9zq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:55.127984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.127979    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c526d837-3a5a-484b-9e2b-996d269bd64d-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:55.198282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.198247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:55.317315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.317224    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:55.627090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.627048    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 17:51:55.680744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.680704    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"
Oct 30 17:51:55.680918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.680863    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8" event={"ID":"c526d837-3a5a-484b-9e2b-996d269bd64d","Type":"ContainerDied","Data":"20efb360899e6460310121e7ebb074bec7e5ee22bba0c9409e1ec41369ed2a45"}
Oct 30 17:51:55.681007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:55.680917    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20efb360899e6460310121e7ebb074bec7e5ee22bba0c9409e1ec41369ed2a45"
Oct 30 17:51:56.470290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.470263    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:51:56.654515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654480    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.654757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654557    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.654757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654706    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.654899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654787    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.654899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654852    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.654899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654856    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:56.655091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.654968    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.655091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655007    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.655091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655082    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.655368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655346    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:56.655458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655425    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:56.655458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655185    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data\") pod \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\" (UID: \"baef6659-67fa-4a2a-9b4b-a8dfccf45f39\") "
Oct 30 17:51:56.655576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655453    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:51:56.655685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.655617    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:56.656208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.656173    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.656208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.656195    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.656376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.656209    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.656376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.656225    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.656376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.656238    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.657548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.657507    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:51:56.659266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.658222    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "kube-api-access-j4bwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:56.659266 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.659235    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:51:56.660805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.660781    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token" pod "baef6659-67fa-4a2a-9b4b-a8dfccf45f39" (UID: "baef6659-67fa-4a2a-9b4b-a8dfccf45f39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:51:56.693047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.692968    5534 generic.go:356] "Generic (PLEG): container finished" podID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerID="4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f" exitCode=0
Oct 30 17:51:56.693187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.693079    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" event={"ID":"baef6659-67fa-4a2a-9b4b-a8dfccf45f39","Type":"ContainerDied","Data":"4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f"}
Oct 30 17:51:56.693187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.693124    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx" event={"ID":"baef6659-67fa-4a2a-9b4b-a8dfccf45f39","Type":"ContainerDied","Data":"d3cf00bda70e1542c113b8d89a18f355a6af86374c141adf19d0c87fd8b2464d"}
Oct 30 17:51:56.693187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.693154    5534 scope.go:122] "RemoveContainer" containerID="4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f"
Oct 30 17:51:56.693316 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.693303    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"
Oct 30 17:51:56.707947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.707921    5534 scope.go:122] "RemoveContainer" containerID="4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f"
Oct 30 17:51:56.708314 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:56.708287    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f\": container with ID starting with 4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f not found: ID does not exist" containerID="4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f"
Oct 30 17:51:56.708382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.708332    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f"} err="failed to get container status \"4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f\": rpc error: code = NotFound desc = could not find container \"4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f\": container with ID starting with 4620b89814e65b136af60f8708610746d20c78de8974f5b203f8ddb0b505d18f not found: ID does not exist"
Oct 30 17:51:56.754207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.754173    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"]
Oct 30 17:51:56.757511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.757482    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.757615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.757518    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.757615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.757532    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.757615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.757547    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bwg\" (UniqueName: \"kubernetes.io/projected/baef6659-67fa-4a2a-9b4b-a8dfccf45f39-kube-api-access-j4bwg\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:51:56.758101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:56.758080    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-qwgbx"]
Oct 30 17:51:56.778667 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:51:56.778607    5534 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaef6659_67fa_4a2a_9b4b_a8dfccf45f39.slice/crio-d3cf00bda70e1542c113b8d89a18f355a6af86374c141adf19d0c87fd8b2464d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaef6659_67fa_4a2a_9b4b_a8dfccf45f39.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 17:51:57.194386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:57.194345    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:57.582486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:57.582394    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 17:51:57.582486 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:57.582476    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 17:51:58.374321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:58.374290    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:51:58.375603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:58.375577    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" path="/var/lib/kubelet/pods/baef6659-67fa-4a2a-9b4b-a8dfccf45f39/volumes"
Oct 30 17:51:58.966217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:51:58.966179    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:00.843460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:00.843419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:02.202024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:02.201981    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"]
Oct 30 17:52:02.339578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:02.339525    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:02.452337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:02.452233    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:02.565098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:02.565060    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:04.208876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:04.208839    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:05.461681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.461603    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"]
Oct 30 17:52:05.469668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.462872    5534 cpu_manager.go:414] "containerMap: removing container" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy"
Oct 30 17:52:05.469982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.469946    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy"
Oct 30 17:52:05.470171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.470156    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c526d837-3a5a-484b-9e2b-996d269bd64d" containerName="revoke-keys"
Oct 30 17:52:05.470271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.470260    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c526d837-3a5a-484b-9e2b-996d269bd64d" containerName="revoke-keys"
Oct 30 17:52:05.470683 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.470662    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="baef6659-67fa-4a2a-9b4b-a8dfccf45f39" containerName="istio-proxy"
Oct 30 17:52:05.470824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.470812    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c526d837-3a5a-484b-9e2b-996d269bd64d" containerName="revoke-keys"
Oct 30 17:52:05.488524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.488493    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.559672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.550573    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.796798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.796707    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgjm\" (UniqueName: \"kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.801188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.801154    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.802365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.797757    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"]
Oct 30 17:52:05.820170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.560342    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:52:05.820170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.560394    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:52:05.904123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.904094    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.904392 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.904373    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgjm\" (UniqueName: \"kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.904756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.904734    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.905018 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.904997    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.939112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.939075    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:05.943656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:05.942785    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgjm\" (UniqueName: \"kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm\") pod \"maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:06.130198 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:06.129892    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:06.295268 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:06.295161    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:06.415002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:06.414899    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:06.550713 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:52:06.550635    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71c4702_7d93_4a13_a396_058547fc0a57.slice/crio-79c7b2a0f8364f49d85319fbd087bd2601a4fe07cd20cd6130d86e2ab3552110 WatchSource:0}: Error finding container 79c7b2a0f8364f49d85319fbd087bd2601a4fe07cd20cd6130d86e2ab3552110: Status 404 returned error can't find the container with id 79c7b2a0f8364f49d85319fbd087bd2601a4fe07cd20cd6130d86e2ab3552110
Oct 30 17:52:06.553861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:06.553832    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"]
Oct 30 17:52:06.889088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:06.889046    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp" event={"ID":"b71c4702-7d93-4a13-a396-058547fc0a57","Type":"ContainerStarted","Data":"79c7b2a0f8364f49d85319fbd087bd2601a4fe07cd20cd6130d86e2ab3552110"}
Oct 30 17:52:07.202732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:07.202605    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerName="payload-processing" containerID="cri-o://3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9" gracePeriod=25
Oct 30 17:52:07.902457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:07.902423    5534 generic.go:356] "Generic (PLEG): container finished" podID="b71c4702-7d93-4a13-a396-058547fc0a57" containerID="bc61465656aac5fc6f176f5012b7849b75def8484f905896ec7c2cc2623fdf2d" exitCode=0
Oct 30 17:52:07.903042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:07.902465    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp" event={"ID":"b71c4702-7d93-4a13-a396-058547fc0a57","Type":"ContainerDied","Data":"bc61465656aac5fc6f176f5012b7849b75def8484f905896ec7c2cc2623fdf2d"}
Oct 30 17:52:08.031401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:08.031368    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"]
Oct 30 17:52:08.035176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:08.035152    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb9053a-bh8wd"]
Oct 30 17:52:08.341940 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:08.341909    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:08.373734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:08.373699    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fe312b-74d3-4040-af1e-d8aa8621ba47" path="/var/lib/kubelet/pods/f1fe312b-74d3-4040-af1e-d8aa8621ba47/volumes"
Oct 30 17:52:09.092015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.091990    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:09.170031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.169938    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm\" (UniqueName: \"kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm\") pod \"b71c4702-7d93-4a13-a396-058547fc0a57\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") "
Oct 30 17:52:09.170257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.170235    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca\") pod \"b71c4702-7d93-4a13-a396-058547fc0a57\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") "
Oct 30 17:52:09.170362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.170330    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca\") pod \"b71c4702-7d93-4a13-a396-058547fc0a57\" (UID: \"b71c4702-7d93-4a13-a396-058547fc0a57\") "
Oct 30 17:52:09.170696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.170618    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca" pod "b71c4702-7d93-4a13-a396-058547fc0a57" (UID: "b71c4702-7d93-4a13-a396-058547fc0a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:09.171297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.171271    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:09.172808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.172776    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm" pod "b71c4702-7d93-4a13-a396-058547fc0a57" (UID: "b71c4702-7d93-4a13-a396-058547fc0a57"). InnerVolumeSpecName "kube-api-access-rbgjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:09.173145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.173117    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca" pod "b71c4702-7d93-4a13-a396-058547fc0a57" (UID: "b71c4702-7d93-4a13-a396-058547fc0a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:52:09.272520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.272487    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c4702-7d93-4a13-a396-058547fc0a57-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:09.272520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.272518    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbgjm\" (UniqueName: \"kubernetes.io/projected/b71c4702-7d93-4a13-a396-058547fc0a57-kube-api-access-rbgjm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:09.920932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.920903    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"
Oct 30 17:52:09.921089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.920934    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp" event={"ID":"b71c4702-7d93-4a13-a396-058547fc0a57","Type":"ContainerDied","Data":"79c7b2a0f8364f49d85319fbd087bd2601a4fe07cd20cd6130d86e2ab3552110"}
Oct 30 17:52:09.921089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:09.920982    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c7b2a0f8364f49d85319fbd087bd2601a4fe07cd20cd6130d86e2ab3552110"
Oct 30 17:52:10.090980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:10.090937    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:11.359830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:11.359744    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:11.718979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:11.718884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:12.411557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:12.411517    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:13.604401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:13.604373    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:15.225384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:15.225353    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:15.246335 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:52:15.246301    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:52:15.930240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:15.930202    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"]
Oct 30 17:52:15.965727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:15.965690    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"]
Oct 30 17:52:16.084927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:16.084068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:16.197923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:16.197788    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:17.015599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:17.015565    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:17.165699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:17.165625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:18.798128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:18.798093    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:19.035586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:19.035549    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"]
Oct 30 17:52:19.047184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:19.047156    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0016b-4fmwd"]
Oct 30 17:52:20.378615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:20.378573    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffb2777-53f9-4a17-acc9-5aeb50c618ee" path="/var/lib/kubelet/pods/dffb2777-53f9-4a17-acc9-5aeb50c618ee/volumes"
Oct 30 17:52:20.498504 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:20.498470    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:20.931156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:20.931115    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerName="payload-processing" containerID="cri-o://f6f19060f2a0dcb04ff5c0c7ba1bc11e8c634b57f4db3c0a18413926a954c922" gracePeriod=25
Oct 30 17:52:20.966276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:20.966240    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerName="payload-pre-processing" containerID="cri-o://356efe66a8b366c2d1ca3ca9ad1bb59ea1aa71b917f43e436809305d61e6a333" gracePeriod=25
Oct 30 17:52:22.000039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.000008    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:22.440495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.440455    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:22.572410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.572369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:22.697543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.697461    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:22.713588 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.713560    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"]
Oct 30 17:52:22.717401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.717369    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-488c67-8r4xp"]
Oct 30 17:52:22.814981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:22.814948    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:23.227955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:23.227924    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:23.729875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:23.729518    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:23.729875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:23.729872    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:24.378460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:24.378427    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71c4702-7d93-4a13-a396-058547fc0a57" path="/var/lib/kubelet/pods/b71c4702-7d93-4a13-a396-058547fc0a57/volumes"
Oct 30 17:52:24.756094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:24.755958    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:24.876320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:24.876286    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:26.607050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:26.607006    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:27.511831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.511803    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:52:27.521363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.521337    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume\") pod \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") "
Oct 30 17:52:27.521535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.521466    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft\" (UniqueName: \"kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft\") pod \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\" (UID: \"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878\") "
Oct 30 17:52:27.521762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.521734    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume" pod "a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" (UID: "a02c7722-8f05-4f8c-ad9b-9c6f1ed63878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:27.522304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.522286    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:27.523864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.523841    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft" pod "a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" (UID: "a02c7722-8f05-4f8c-ad9b-9c6f1ed63878"). InnerVolumeSpecName "kube-api-access-vj8ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:27.624478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.624445    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8ft\" (UniqueName: \"kubernetes.io/projected/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878-kube-api-access-vj8ft\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:27.711082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:27.711048    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:28.085447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.085414    5534 generic.go:356] "Generic (PLEG): container finished" podID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerID="3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9" exitCode=0
Oct 30 17:52:28.085691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.085466    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"
Oct 30 17:52:28.085691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.085491    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" event={"ID":"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878","Type":"ContainerDied","Data":"3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9"}
Oct 30 17:52:28.085691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.085569    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2" event={"ID":"a02c7722-8f05-4f8c-ad9b-9c6f1ed63878","Type":"ContainerDied","Data":"cf8c251fe9dfd782a05a0d0b1aa34eeabfb9126359d6c52e51c1a3defe5fe65c"}
Oct 30 17:52:28.085691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.085598    5534 scope.go:122] "RemoveContainer" containerID="3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9"
Oct 30 17:52:28.100163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.100137    5534 scope.go:122] "RemoveContainer" containerID="3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9"
Oct 30 17:52:28.100504 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:52:28.100476    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9\": container with ID starting with 3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9 not found: ID does not exist" containerID="3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9"
Oct 30 17:52:28.100628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.100511    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9"} err="failed to get container status \"3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9\": rpc error: code = NotFound desc = could not find container \"3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9\": container with ID starting with 3a902672f6a364bef4394b2107259aca42858b29493fe4567aee49c079b236f9 not found: ID does not exist"
Oct 30 17:52:28.139611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.139583    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"]
Oct 30 17:52:28.152077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.152053    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1c345-989c5f5db-8sgx2"]
Oct 30 17:52:28.374863 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.374830    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" path="/var/lib/kubelet/pods/a02c7722-8f05-4f8c-ad9b-9c6f1ed63878/volumes"
Oct 30 17:52:28.486022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:28.485987    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:29.951347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:29.951312    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:30.070378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.070339    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:30.339968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.339945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:30.457280 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.457247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:30.849038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.849003    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"]
Oct 30 17:52:30.849855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.849834    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerName="payload-processing"
Oct 30 17:52:30.849855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.849852    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerName="payload-processing"
Oct 30 17:52:30.850080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.849865    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b71c4702-7d93-4a13-a396-058547fc0a57" containerName="revoke-keys"
Oct 30 17:52:30.850080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.849872    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71c4702-7d93-4a13-a396-058547fc0a57" containerName="revoke-keys"
Oct 30 17:52:30.850080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.849994    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02c7722-8f05-4f8c-ad9b-9c6f1ed63878" containerName="payload-processing"
Oct 30 17:52:30.850080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.850006    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71c4702-7d93-4a13-a396-058547fc0a57" containerName="revoke-keys"
Oct 30 17:52:30.855614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.855587    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:30.876301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.876272    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"]
Oct 30 17:52:30.969173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.969135    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:52:30.969667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.969396    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:52:30.989572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.989527    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qbw\" (UniqueName: \"kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:30.989572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.989590    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:30.989964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:30.989943    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.091921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.091881    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.092149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.092025    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qbw\" (UniqueName: \"kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.092149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.092070    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.092954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.092930    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.094452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.094429    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.115531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.115469    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qbw\" (UniqueName: \"kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw\") pod \"maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.173600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.173564    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:31.490461 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:52:31.490426    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4760578f_f08d_40d2_b990_efab5dbb4746.slice/crio-5255f8f3bee2c8e92071659eb0d9518e590dd1d5d305c148d3e75828ea1ec182 WatchSource:0}: Error finding container 5255f8f3bee2c8e92071659eb0d9518e590dd1d5d305c148d3e75828ea1ec182: Status 404 returned error can't find the container with id 5255f8f3bee2c8e92071659eb0d9518e590dd1d5d305c148d3e75828ea1ec182
Oct 30 17:52:31.492517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:31.492459    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"]
Oct 30 17:52:32.142675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:32.141240    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq" event={"ID":"4760578f-f08d-40d2-b990-efab5dbb4746","Type":"ContainerStarted","Data":"f4e80ccde5b22160f077856787f6a7ba3cb7df1d4913288be6a84ecae8a5efce"}
Oct 30 17:52:32.142675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:32.141316    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq" event={"ID":"4760578f-f08d-40d2-b990-efab5dbb4746","Type":"ContainerStarted","Data":"5255f8f3bee2c8e92071659eb0d9518e590dd1d5d305c148d3e75828ea1ec182"}
Oct 30 17:52:32.183283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:32.183231    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq" podStartSLOduration=2.183215933 podStartE2EDuration="2.183215933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:32.182462347 +0000 UTC m=+2357.944506165" watchObservedRunningTime="2026-10-30 17:52:32.183215933 +0000 UTC m=+2357.945259753"
Oct 30 17:52:32.398655 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:32.398605    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:33.155749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:33.155714    5534 generic.go:356] "Generic (PLEG): container finished" podID="4760578f-f08d-40d2-b990-efab5dbb4746" containerID="f4e80ccde5b22160f077856787f6a7ba3cb7df1d4913288be6a84ecae8a5efce" exitCode=0
Oct 30 17:52:33.156148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:33.155758    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq" event={"ID":"4760578f-f08d-40d2-b990-efab5dbb4746","Type":"ContainerDied","Data":"f4e80ccde5b22160f077856787f6a7ba3cb7df1d4913288be6a84ecae8a5efce"}
Oct 30 17:52:33.729743 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:33.729698    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:33.729945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:33.729825    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:34.224791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.224758    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:34.361028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.360995    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:34.448479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.448391    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca\") pod \"4760578f-f08d-40d2-b990-efab5dbb4746\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") "
Oct 30 17:52:34.448688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.448499    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw\" (UniqueName: \"kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw\") pod \"4760578f-f08d-40d2-b990-efab5dbb4746\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") "
Oct 30 17:52:34.448783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.448726    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca\") pod \"4760578f-f08d-40d2-b990-efab5dbb4746\" (UID: \"4760578f-f08d-40d2-b990-efab5dbb4746\") "
Oct 30 17:52:34.448783 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.448765    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca" pod "4760578f-f08d-40d2-b990-efab5dbb4746" (UID: "4760578f-f08d-40d2-b990-efab5dbb4746"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:34.449440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.449414    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:34.458255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.458221    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw" pod "4760578f-f08d-40d2-b990-efab5dbb4746" (UID: "4760578f-f08d-40d2-b990-efab5dbb4746"). InnerVolumeSpecName "kube-api-access-r2qbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:34.458397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.458327    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca" pod "4760578f-f08d-40d2-b990-efab5dbb4746" (UID: "4760578f-f08d-40d2-b990-efab5dbb4746"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:52:34.551174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.551137    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qbw\" (UniqueName: \"kubernetes.io/projected/4760578f-f08d-40d2-b990-efab5dbb4746-kube-api-access-r2qbw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:34.551174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:34.551170    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4760578f-f08d-40d2-b990-efab5dbb4746-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:35.174805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:35.174774    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"
Oct 30 17:52:35.174997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:35.174808    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq" event={"ID":"4760578f-f08d-40d2-b990-efab5dbb4746","Type":"ContainerDied","Data":"5255f8f3bee2c8e92071659eb0d9518e590dd1d5d305c148d3e75828ea1ec182"}
Oct 30 17:52:35.174997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:35.174863    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5255f8f3bee2c8e92071659eb0d9518e590dd1d5d305c148d3e75828ea1ec182"
Oct 30 17:52:35.898391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:35.898340    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:37.356384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:37.356346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:37.919701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:37.919668    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:38.093429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:38.093388    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:38.157996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:38.157845    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"]
Oct 30 17:52:38.168758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:38.168636    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1c345-55dl8"]
Oct 30 17:52:38.281297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:38.281208    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:38.376278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:38.376240    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c526d837-3a5a-484b-9e2b-996d269bd64d" path="/var/lib/kubelet/pods/c526d837-3a5a-484b-9e2b-996d269bd64d/volumes"
Oct 30 17:52:38.670419 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:38.670384    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:40.186247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:40.186194    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:41.261664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.253379    5534 generic.go:356] "Generic (PLEG): container finished" podID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerID="356efe66a8b366c2d1ca3ca9ad1bb59ea1aa71b917f43e436809305d61e6a333" exitCode=0
Oct 30 17:52:41.261664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.253549    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" event={"ID":"d26c387f-250d-4e59-8106-b25975fc9f3f","Type":"ContainerDied","Data":"356efe66a8b366c2d1ca3ca9ad1bb59ea1aa71b917f43e436809305d61e6a333"}
Oct 30 17:52:41.261664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.255017    5534 generic.go:356] "Generic (PLEG): container finished" podID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerID="f6f19060f2a0dcb04ff5c0c7ba1bc11e8c634b57f4db3c0a18413926a954c922" exitCode=0
Oct 30 17:52:41.261664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.255155    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" event={"ID":"25dd3c0c-58f4-4e48-bffa-96a9a14c5157","Type":"ContainerDied","Data":"f6f19060f2a0dcb04ff5c0c7ba1bc11e8c634b57f4db3c0a18413926a954c922"}
Oct 30 17:52:41.293431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.293388    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:52:41.351225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.351191    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:52:41.353882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.353857    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"]
Oct 30 17:52:41.398672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.392768    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm\" (UniqueName: \"kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm\") pod \"d26c387f-250d-4e59-8106-b25975fc9f3f\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") "
Oct 30 17:52:41.398672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.392902    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume\") pod \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") "
Oct 30 17:52:41.398672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.392964    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q\" (UniqueName: \"kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q\") pod \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\" (UID: \"25dd3c0c-58f4-4e48-bffa-96a9a14c5157\") "
Oct 30 17:52:41.398672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.393207    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume\") pod \"d26c387f-250d-4e59-8106-b25975fc9f3f\" (UID: \"d26c387f-250d-4e59-8106-b25975fc9f3f\") "
Oct 30 17:52:41.402004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.400133    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume" pod "d26c387f-250d-4e59-8106-b25975fc9f3f" (UID: "d26c387f-250d-4e59-8106-b25975fc9f3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:41.402004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.400790    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume" pod "25dd3c0c-58f4-4e48-bffa-96a9a14c5157" (UID: "25dd3c0c-58f4-4e48-bffa-96a9a14c5157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:41.408200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.406117    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm" pod "d26c387f-250d-4e59-8106-b25975fc9f3f" (UID: "d26c387f-250d-4e59-8106-b25975fc9f3f"). InnerVolumeSpecName "kube-api-access-2brgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:41.408372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.408355    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q" pod "25dd3c0c-58f4-4e48-bffa-96a9a14c5157" (UID: "25dd3c0c-58f4-4e48-bffa-96a9a14c5157"). InnerVolumeSpecName "kube-api-access-fgw9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:41.453409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.453366    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:41.495880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.495789    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c387f-250d-4e59-8106-b25975fc9f3f-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:41.496057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.496040    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2brgm\" (UniqueName: \"kubernetes.io/projected/d26c387f-250d-4e59-8106-b25975fc9f3f-kube-api-access-2brgm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:41.496138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.496061    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:41.496138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.496076    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgw9q\" (UniqueName: \"kubernetes.io/projected/25dd3c0c-58f4-4e48-bffa-96a9a14c5157-kube-api-access-fgw9q\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:41.568670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.567172    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:41.918820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:41.918790    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:42.265359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.265260    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb" event={"ID":"d26c387f-250d-4e59-8106-b25975fc9f3f","Type":"ContainerDied","Data":"49c694bc87618dc700542e846f4f0da1d5ea266db5734442dce645d1c950d225"}
Oct 30 17:52:42.265359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.265272    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"
Oct 30 17:52:42.265359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.265342    5534 scope.go:122] "RemoveContainer" containerID="356efe66a8b366c2d1ca3ca9ad1bb59ea1aa71b917f43e436809305d61e6a333"
Oct 30 17:52:42.267527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.267442    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"
Oct 30 17:52:42.267527 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.267475    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw" event={"ID":"25dd3c0c-58f4-4e48-bffa-96a9a14c5157","Type":"ContainerDied","Data":"1894ad5eb03d69de9e673895de13e391f6030c5353a8612a5d1c64e161d02dbb"}
Oct 30 17:52:42.279035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.279018    5534 scope.go:122] "RemoveContainer" containerID="f6f19060f2a0dcb04ff5c0c7ba1bc11e8c634b57f4db3c0a18413926a954c922"
Oct 30 17:52:42.318906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.318870    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"]
Oct 30 17:52:42.322778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.322753    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-488c67-664dfb6b59-gg5hb"]
Oct 30 17:52:42.374850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.374816    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" path="/var/lib/kubelet/pods/d26c387f-250d-4e59-8106-b25975fc9f3f/volumes"
Oct 30 17:52:42.376513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.376487    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"]
Oct 30 17:52:42.380988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:42.380967    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-488c67-79fc8b6448-8dvzw"]
Oct 30 17:52:43.617997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:43.617965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:44.389317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:44.389275    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" path="/var/lib/kubelet/pods/25dd3c0c-58f4-4e48-bffa-96a9a14c5157/volumes"
Oct 30 17:52:45.017005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:45.016971    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:45.133564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:45.133532    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:45.246830 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:52:45.246788    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:52:45.336710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:45.336608    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"]
Oct 30 17:52:45.336934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:45.336907    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy" containerID="cri-o://df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b" gracePeriod=30
Oct 30 17:52:45.364719 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:45.361655    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:45.478665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:45.478552    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:46.195880 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:46.195844    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:46.354816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:46.354766    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerName="payload-pre-processing" containerID="cri-o://dd4eb1ca1cd9ee0171262ccd4e573cc0a121e937b7c56a2699c852c162725f45" gracePeriod=25
Oct 30 17:52:46.375126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:46.375094    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:46.695199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:46.695162    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:46.833294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:46.828944    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:47.181809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:47.181777    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:48.936111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:48.936076    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:49.046976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.046941    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:49.738853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.738817    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"]
Oct 30 17:52:49.739977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.739943    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerName="payload-pre-processing"
Oct 30 17:52:49.739977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.739976    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerName="payload-pre-processing"
Oct 30 17:52:49.740209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.739998    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4760578f-f08d-40d2-b990-efab5dbb4746" containerName="revoke-keys"
Oct 30 17:52:49.740209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.740006    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4760578f-f08d-40d2-b990-efab5dbb4746" containerName="revoke-keys"
Oct 30 17:52:49.740209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.740089    5534 cpu_manager.go:414] "containerMap: removing container" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerName="payload-processing"
Oct 30 17:52:49.740209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.740097    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerName="payload-processing"
Oct 30 17:52:49.740360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.740299    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="25dd3c0c-58f4-4e48-bffa-96a9a14c5157" containerName="payload-processing"
Oct 30 17:52:49.740360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.740322    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26c387f-250d-4e59-8106-b25975fc9f3f" containerName="payload-pre-processing"
Oct 30 17:52:49.740360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.740334    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4760578f-f08d-40d2-b990-efab5dbb4746" containerName="revoke-keys"
Oct 30 17:52:49.746236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.746216    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.780145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.780117    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:52:49.780330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.780187    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:52:49.783886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.783846    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"]
Oct 30 17:52:49.868217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.868175    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.868377 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.868236    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdh4\" (UniqueName: \"kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.868695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.868664    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.970335 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.970294    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.970820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.970383    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.970820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.970423    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdh4\" (UniqueName: \"kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.971135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.971107    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.972984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.972954    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:49.994930 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:49.994860    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdh4\" (UniqueName: \"kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4\") pod \"maas-api-revoke-sub-9e19b616-ktjmv\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:50.070595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.070557    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:50.399093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.399052    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"]
Oct 30 17:52:50.418274 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:52:50.418239    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd267fc23_db31_4a8a_8cff_42cddb43a3f7.slice/crio-0b33a77c4ca4b6291e7acb87bfc82c0831cdd244828bf4ff09e1f82d86e09ee9 WatchSource:0}: Error finding container 0b33a77c4ca4b6291e7acb87bfc82c0831cdd244828bf4ff09e1f82d86e09ee9: Status 404 returned error can't find the container with id 0b33a77c4ca4b6291e7acb87bfc82c0831cdd244828bf4ff09e1f82d86e09ee9
Oct 30 17:52:50.649336 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.649306    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:52:50.783659 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.783605    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.783858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.783822    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.783858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.783854    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.783978    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784023    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784072    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784206    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784274    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784305    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs\") pod \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\" (UID: \"105ab0a0-4585-4c94-8eb6-c789cd5b00a1\") "
Oct 30 17:52:50.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784403    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:52:50.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784491    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:50.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.784684    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:52:50.785271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.785080    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.785271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.785105    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.785271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.785121    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.785511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.785350    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:52:50.785613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.785577    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:52:50.868684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.865804    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:52:50.868684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.865950    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "kube-api-access-5nnl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:50.868684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.866028    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:52:50.868684 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.867890    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token" pod "105ab0a0-4585-4c94-8eb6-c789cd5b00a1" (UID: "105ab0a0-4585-4c94-8eb6-c789cd5b00a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:50.891715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.890951    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.891715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.890994    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.891715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.891011    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.891715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.891025    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.891715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.891037    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:50.891715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:50.891049    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nnl6\" (UniqueName: \"kubernetes.io/projected/105ab0a0-4585-4c94-8eb6-c789cd5b00a1-kube-api-access-5nnl6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:51.063596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.058318    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:51.367757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.359230    5534 generic.go:356] "Generic (PLEG): container finished" podID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerID="df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b" exitCode=0
Oct 30 17:52:51.367757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.359416    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" event={"ID":"105ab0a0-4585-4c94-8eb6-c789cd5b00a1","Type":"ContainerDied","Data":"df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b"}
Oct 30 17:52:51.367757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.359471    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6" event={"ID":"105ab0a0-4585-4c94-8eb6-c789cd5b00a1","Type":"ContainerDied","Data":"5793fbec0b679b603447765bfaa8100a09de668c0b2c0d0adf1ee2c6b3237130"}
Oct 30 17:52:51.367757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.359501    5534 scope.go:122] "RemoveContainer" containerID="df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b"
Oct 30 17:52:51.367757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.359709    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"
Oct 30 17:52:51.373001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.371210    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv" event={"ID":"d267fc23-db31-4a8a-8cff-42cddb43a3f7","Type":"ContainerStarted","Data":"9237d4bc3e3cfb952e9392666bd2f4d821c6cd2040fe12f0110d0563dbb6c1b7"}
Oct 30 17:52:51.373001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.371275    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv" event={"ID":"d267fc23-db31-4a8a-8cff-42cddb43a3f7","Type":"ContainerStarted","Data":"0b33a77c4ca4b6291e7acb87bfc82c0831cdd244828bf4ff09e1f82d86e09ee9"}
Oct 30 17:52:51.406050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.406029    5534 scope.go:122] "RemoveContainer" containerID="f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"
Oct 30 17:52:51.424019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.423996    5534 scope.go:122] "RemoveContainer" containerID="df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b"
Oct 30 17:52:51.424420 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:52:51.424386    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b\": container with ID starting with df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b not found: ID does not exist" containerID="df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b"
Oct 30 17:52:51.424546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.424430    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b"} err="failed to get container status \"df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b\": rpc error: code = NotFound desc = could not find container \"df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b\": container with ID starting with df45f524fb76d0cfa57fb63da935ce37bc2b27190f4c9f6718afe03a4088d99b not found: ID does not exist"
Oct 30 17:52:51.424546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.424453    5534 scope.go:122] "RemoveContainer" containerID="f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"
Oct 30 17:52:51.425030 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:52:51.425005    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a\": container with ID starting with f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a not found: ID does not exist" containerID="f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"
Oct 30 17:52:51.425245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.425037    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a"} err="failed to get container status \"f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a\": rpc error: code = NotFound desc = could not find container \"f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a\": container with ID starting with f6d3ae34a1d163f10374cc01c181d3a3846bfff5ce5b72b4cd2e002bd598c41a not found: ID does not exist"
Oct 30 17:52:51.428678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.427623    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"]
Oct 30 17:52:51.431371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.431342    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b1c345-openshift-default-7cb6866669-rshr6"]
Oct 30 17:52:51.470090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.470038    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv" podStartSLOduration=2.470022368 podStartE2EDuration="2.470022368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:51.468512887 +0000 UTC m=+2377.230556701" watchObservedRunningTime="2026-10-30 17:52:51.470022368 +0000 UTC m=+2377.232066194"
Oct 30 17:52:51.580737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:51.580679    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:52:52.425750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:52.425712    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" path="/var/lib/kubelet/pods/105ab0a0-4585-4c94-8eb6-c789cd5b00a1/volumes"
Oct 30 17:52:52.427242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:52.427208    5534 generic.go:356] "Generic (PLEG): container finished" podID="d267fc23-db31-4a8a-8cff-42cddb43a3f7" containerID="9237d4bc3e3cfb952e9392666bd2f4d821c6cd2040fe12f0110d0563dbb6c1b7" exitCode=0
Oct 30 17:52:52.427425 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:52.427406    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv" event={"ID":"d267fc23-db31-4a8a-8cff-42cddb43a3f7","Type":"ContainerDied","Data":"9237d4bc3e3cfb952e9392666bd2f4d821c6cd2040fe12f0110d0563dbb6c1b7"}
Oct 30 17:52:52.931091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:52.931040    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:53.824021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.823985    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:53.919771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.919742    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca\") pod \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") "
Oct 30 17:52:53.927849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.927824    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca\") pod \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") "
Oct 30 17:52:53.928070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.928054    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4\" (UniqueName: \"kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4\") pod \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\" (UID: \"d267fc23-db31-4a8a-8cff-42cddb43a3f7\") "
Oct 30 17:52:53.933408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.933384    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca" pod "d267fc23-db31-4a8a-8cff-42cddb43a3f7" (UID: "d267fc23-db31-4a8a-8cff-42cddb43a3f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:52:53.935802 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.934564    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca" pod "d267fc23-db31-4a8a-8cff-42cddb43a3f7" (UID: "d267fc23-db31-4a8a-8cff-42cddb43a3f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:52:53.951220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:53.951184    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4" pod "d267fc23-db31-4a8a-8cff-42cddb43a3f7" (UID: "d267fc23-db31-4a8a-8cff-42cddb43a3f7"). InnerVolumeSpecName "kube-api-access-vxdh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:52:54.034202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.034103    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:54.034202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.034132    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267fc23-db31-4a8a-8cff-42cddb43a3f7-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:54.034202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.034147    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdh4\" (UniqueName: \"kubernetes.io/projected/d267fc23-db31-4a8a-8cff-42cddb43a3f7-kube-api-access-vxdh4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:52:54.449865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.449001    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv" event={"ID":"d267fc23-db31-4a8a-8cff-42cddb43a3f7","Type":"ContainerDied","Data":"0b33a77c4ca4b6291e7acb87bfc82c0831cdd244828bf4ff09e1f82d86e09ee9"}
Oct 30 17:52:54.449865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.449061    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b33a77c4ca4b6291e7acb87bfc82c0831cdd244828bf4ff09e1f82d86e09ee9"
Oct 30 17:52:54.449865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.449126    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"
Oct 30 17:52:54.794728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:54.794271    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:57.122699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:57.120373    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:52:58.905356 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:52:58.905315    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:00.803567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:00.803532    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:01.084106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:01.083983    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:01.202406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:01.202346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:01.580114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:01.580079    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:53:02.839526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:02.839475    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:02.976119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:02.976068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:03.013964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.013916    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"]
Oct 30 17:53:03.015193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015161    5534 cpu_manager.go:414] "containerMap: removing container" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy"
Oct 30 17:53:03.015193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015193    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy"
Oct 30 17:53:03.015408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015233    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d267fc23-db31-4a8a-8cff-42cddb43a3f7" containerName="revoke-keys"
Oct 30 17:53:03.015408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015242    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d267fc23-db31-4a8a-8cff-42cddb43a3f7" containerName="revoke-keys"
Oct 30 17:53:03.015408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015318    5534 cpu_manager.go:414] "containerMap: removing container" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy"
Oct 30 17:53:03.015408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015326    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy"
Oct 30 17:53:03.015675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015501    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy"
Oct 30 17:53:03.015675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.015523    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d267fc23-db31-4a8a-8cff-42cddb43a3f7" containerName="revoke-keys"
Oct 30 17:53:03.072631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.071884    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"]
Oct 30 17:53:03.072631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.072031    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.111693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.111596    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-23c1e8\""
Oct 30 17:53:03.112461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.112415    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-23c1e8-dockercfg-hmrfv\""
Oct 30 17:53:03.116528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.116498    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"]
Oct 30 17:53:03.117866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.117843    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="105ab0a0-4585-4c94-8eb6-c789cd5b00a1" containerName="istio-proxy"
Oct 30 17:53:03.179355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.179102    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrdd\" (UniqueName: \"kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd\") pod \"payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.191030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.190994    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"]
Oct 30 17:53:03.191383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.191365    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.201542 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.201490    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.323670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.322550    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrdd\" (UniqueName: \"kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd\") pod \"payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.323670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.322722    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw98k\" (UniqueName: \"kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k\") pod \"payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.323670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.322877    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.323670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.322940    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.343758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.324116    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.369450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.369370    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrdd\" (UniqueName: \"kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd\") pod \"payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.424079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.424039    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.424696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.424426    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw98k\" (UniqueName: \"kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k\") pod \"payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.425073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.424950    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.450652 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.450600    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw98k\" (UniqueName: \"kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k\") pod \"payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.474652 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.474595    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:03.585110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.584426    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:53:03.585110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.584499    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:53:03.674274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.674190    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:03.838613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:03.838388    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"]
Oct 30 17:53:03.838613 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:03.838541    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5ecc83_b741_41e6_b818_9133b1feb346.slice/crio-2ff8946a30783140c212fcb99222283e0bf99b6993a58f38ba94fd975edd55cf WatchSource:0}: Error finding container 2ff8946a30783140c212fcb99222283e0bf99b6993a58f38ba94fd975edd55cf: Status 404 returned error can't find the container with id 2ff8946a30783140c212fcb99222283e0bf99b6993a58f38ba94fd975edd55cf
Oct 30 17:53:04.007967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.007936    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"]
Oct 30 17:53:04.009354 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:04.009329    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c17f33_9fb3_42c9_953d_5cd384d1d3a7.slice/crio-f4f368f337ae5656c613857ccbd3926f307bead3eae544ecb22b6dd6c1f96bc6 WatchSource:0}: Error finding container f4f368f337ae5656c613857ccbd3926f307bead3eae544ecb22b6dd6c1f96bc6: Status 404 returned error can't find the container with id f4f368f337ae5656c613857ccbd3926f307bead3eae544ecb22b6dd6c1f96bc6
Oct 30 17:53:04.591873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.591832    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" event={"ID":"38c17f33-9fb3-42c9-953d-5cd384d1d3a7","Type":"ContainerStarted","Data":"131fbc09f84bb6c7926e9533a1b44db550ae9239b0dc28a47d040f97d9d5209b"}
Oct 30 17:53:04.592190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.591906    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:04.592190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.591929    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" event={"ID":"38c17f33-9fb3-42c9-953d-5cd384d1d3a7","Type":"ContainerStarted","Data":"f4f368f337ae5656c613857ccbd3926f307bead3eae544ecb22b6dd6c1f96bc6"}
Oct 30 17:53:04.593914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.593873    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" event={"ID":"df5ecc83-b741-41e6-b818-9133b1feb346","Type":"ContainerStarted","Data":"21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10"}
Oct 30 17:53:04.594050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.593945    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" event={"ID":"df5ecc83-b741-41e6-b818-9133b1feb346","Type":"ContainerStarted","Data":"2ff8946a30783140c212fcb99222283e0bf99b6993a58f38ba94fd975edd55cf"}
Oct 30 17:53:04.594050 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.593979    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:04.596214 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.596188    5534 generic.go:356] "Generic (PLEG): container finished" podID="4ed38c99-08e0-476e-aa19-6602e5073655" containerID="5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf" exitCode=0
Oct 30 17:53:04.596379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.596245    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" event={"ID":"4ed38c99-08e0-476e-aa19-6602e5073655","Type":"ContainerDied","Data":"5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"}
Oct 30 17:53:04.596625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.596607    5534 scope.go:122] "RemoveContainer" containerID="5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"
Oct 30 17:53:04.598875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.598833    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:04.598988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.598921    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:04.599061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.598990    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:04.635495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.635451    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" podStartSLOduration=1.6354364449999998 podStartE2EDuration="1.635436445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:04.63144812 +0000 UTC m=+2390.393491936" watchObservedRunningTime="2026-10-30 17:53:04.635436445 +0000 UTC m=+2390.397480277"
Oct 30 17:53:04.680133 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.679625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:04.722412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:04.722355    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" podStartSLOduration=2.722339012 podStartE2EDuration="2.722339012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:04.720682574 +0000 UTC m=+2390.482726385" watchObservedRunningTime="2026-10-30 17:53:04.722339012 +0000 UTC m=+2390.484382819"
Oct 30 17:53:05.609974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.609935    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" event={"ID":"4ed38c99-08e0-476e-aa19-6602e5073655","Type":"ContainerStarted","Data":"7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66"}
Oct 30 17:53:05.612003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.611980    5534 generic.go:356] "Generic (PLEG): container finished" podID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerID="ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5" exitCode=0
Oct 30 17:53:05.612109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.612044    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" event={"ID":"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9","Type":"ContainerDied","Data":"ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"}
Oct 30 17:53:05.612913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.612897    5534 scope.go:122] "RemoveContainer" containerID="ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"
Oct 30 17:53:05.614637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.614601    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:05.614743 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.614710    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:05.614786 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:05.614745    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:06.518265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.518188    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:06.520624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.520493    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:53:06.520624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.520534    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:53:06.520624 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.520555    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:53:06.644567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.644534    5534 generic.go:356] "Generic (PLEG): container finished" podID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerID="dd4eb1ca1cd9ee0171262ccd4e573cc0a121e937b7c56a2699c852c162725f45" exitCode=0
Oct 30 17:53:06.648949 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.644762    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" event={"ID":"3c82e6b3-5b85-4371-977d-0aa90358fac2","Type":"ContainerDied","Data":"dd4eb1ca1cd9ee0171262ccd4e573cc0a121e937b7c56a2699c852c162725f45"}
Oct 30 17:53:06.678988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.678954    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" event={"ID":"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9","Type":"ContainerStarted","Data":"a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1"}
Oct 30 17:53:06.827666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.827328    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:53:06.942787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.942684    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4\" (UniqueName: \"kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4\") pod \"3c82e6b3-5b85-4371-977d-0aa90358fac2\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") "
Oct 30 17:53:06.943150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.942974    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume\") pod \"3c82e6b3-5b85-4371-977d-0aa90358fac2\" (UID: \"3c82e6b3-5b85-4371-977d-0aa90358fac2\") "
Oct 30 17:53:06.943981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.943953    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume" pod "3c82e6b3-5b85-4371-977d-0aa90358fac2" (UID: "3c82e6b3-5b85-4371-977d-0aa90358fac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:06.948277 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.948252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4" pod "3c82e6b3-5b85-4371-977d-0aa90358fac2" (UID: "3c82e6b3-5b85-4371-977d-0aa90358fac2"). InnerVolumeSpecName "kube-api-access-5v5m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:06.964678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.960017    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:53:06.964992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.964964    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 17:53:06.965182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:06.965155    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 17:53:07.044916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.044889    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c82e6b3-5b85-4371-977d-0aa90358fac2-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:07.045107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.045096    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5m4\" (UniqueName: \"kubernetes.io/projected/3c82e6b3-5b85-4371-977d-0aa90358fac2-kube-api-access-5v5m4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:07.520939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.520904    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:53:07.521140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.520960    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:53:07.723103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.723075    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"
Oct 30 17:53:07.723631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.723188    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj" event={"ID":"3c82e6b3-5b85-4371-977d-0aa90358fac2","Type":"ContainerDied","Data":"ac5b77f969f9c774d5a0ab4fe39dda0b2ef560cb93d06f9f523557f904916627"}
Oct 30 17:53:07.723631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.723272    5534 scope.go:122] "RemoveContainer" containerID="dd4eb1ca1cd9ee0171262ccd4e573cc0a121e937b7c56a2699c852c162725f45"
Oct 30 17:53:07.813953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.813856    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"]
Oct 30 17:53:07.822671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.822609    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-303ba5-749756bd6c-gf6hj"]
Oct 30 17:53:07.974668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.974453    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 17:53:07.974668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:07.974510    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 17:53:08.404665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:08.404485    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" path="/var/lib/kubelet/pods/3c82e6b3-5b85-4371-977d-0aa90358fac2/volumes"
Oct 30 17:53:08.524667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:08.523699    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:53:08.524667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:08.523760    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:53:08.896294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:08.896258    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:08.960415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:08.960379    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 17:53:08.960748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:08.960712    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 17:53:09.521769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:09.521387    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:53:09.521769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:09.521439    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:53:09.961201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:09.961165    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 17:53:09.962360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:09.962313    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 17:53:10.139953 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.139899    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:10.521152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.521120    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:53:10.521402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.521381    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:53:10.611510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.611477    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:10.964846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.964381    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 17:53:10.964846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.964450    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 17:53:10.978675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:10.977887    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:11.048228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.048196    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"]
Oct 30 17:53:11.049029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.049009    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerName="payload-pre-processing"
Oct 30 17:53:11.049206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.049193    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerName="payload-pre-processing"
Oct 30 17:53:11.049463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.049454    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c82e6b3-5b85-4371-977d-0aa90358fac2" containerName="payload-pre-processing"
Oct 30 17:53:11.133716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.130262    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.137903    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138112 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138081    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138179    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138244    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138341    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138369    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138405    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138484    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.138592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.138564    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn22z\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.139813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.139780    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:11.140447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.140419    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"]
Oct 30 17:53:11.152603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.152579    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a23c98-openshift-default-dockercfg-g2m5h\""
Oct 30 17:53:11.361471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.337614    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.361862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.361832    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.362048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.362028    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.362184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.362167    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.362365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.362350    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.362547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.362526    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn22z\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.362739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.362719    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.363061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.363038    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.363208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.363191    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.363886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.363851    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.364752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.364731    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.365111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.365092    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.384485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.384330    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.401709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.399755    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.450811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.436377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:11.450811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.436921    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.450811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.437988    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.450811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.442272    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.475667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.472403    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn22z\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.520687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.520654    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:53:11.520887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.520715    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:53:11.587853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.587553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:11.759718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.757012    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:11.977670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.975003    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 17:53:11.977670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:11.975065    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 17:53:12.119670 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:12.119617    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bfc8c2_a073_4c16_b984_282c82d720e5.slice/crio-99e7a7b86dace30f73e2cb36d5090f90abd2bc18d1b21d1c41a848646e17ed6b WatchSource:0}: Error finding container 99e7a7b86dace30f73e2cb36d5090f90abd2bc18d1b21d1c41a848646e17ed6b: Status 404 returned error can't find the container with id 99e7a7b86dace30f73e2cb36d5090f90abd2bc18d1b21d1c41a848646e17ed6b
Oct 30 17:53:12.119791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.119734    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"]
Oct 30 17:53:12.122376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.122345    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:12.122445 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.122413    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:12.122480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.122442    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:12.519350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.519267    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:53:12.527222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.527188    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:53:12.528245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.528207    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:53:12.528914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.528888    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 17:53:12.807920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.807824    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" event={"ID":"07bfc8c2-a073-4c16-b984-282c82d720e5","Type":"ContainerStarted","Data":"e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"}
Oct 30 17:53:12.807920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.807920    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" event={"ID":"07bfc8c2-a073-4c16-b984-282c82d720e5","Type":"ContainerStarted","Data":"99e7a7b86dace30f73e2cb36d5090f90abd2bc18d1b21d1c41a848646e17ed6b"}
Oct 30 17:53:12.852756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.852688    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podStartSLOduration=1.852624686 podStartE2EDuration="1.852624686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:12.851795528 +0000 UTC m=+2398.613839357" watchObservedRunningTime="2026-10-30 17:53:12.852624686 +0000 UTC m=+2398.614668500"
Oct 30 17:53:12.964035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.964003    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:53:12.965190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:12.965163    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 17:53:13.010770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:13.010745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:13.759490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:13.759462    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:13.766280 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:13.766255    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:13.816183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:13.816153    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:13.816830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:13.816810    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:53:14.585537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:14.585493    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.117583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.117546    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.244956 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:53:15.244889    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:53:15.343901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.343871    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.474013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.473937    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"]
Oct 30 17:53:15.495614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.495578    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-303ba5-t82gq"]
Oct 30 17:53:15.583241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.583183    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.615344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.615318    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 17:53:15.615774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.615424    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 17:53:15.655857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.655284    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"]
Oct 30 17:53:15.661753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.661722    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.681081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.681041    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:15.681233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.681106    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.681321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.681287    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:53:15.681321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.681304    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfph\" (UniqueName: \"kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.681457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.681400    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.687329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.686610    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"]
Oct 30 17:53:15.723856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.723801    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.784887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.784273    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.785322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.785169    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfph\" (UniqueName: \"kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.785322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.785278    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.786095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.786072    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.787345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.787320    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.809865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.809840    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfph\" (UniqueName: \"kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph\") pod \"maas-api-revoke-sub-14abcaa3-swz57\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:15.836429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.836394    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.952574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.952544    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:15.987262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:15.987232    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:16.298492 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.298417    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"]
Oct 30 17:53:16.303099 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:16.303056    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107b59fd_c805_400b_b944_35d563b1020f.slice/crio-7ea4e4cf4a3880d309d16c76630a9a35bada792ac800dc3b80a6eb6405fb8bf6 WatchSource:0}: Error finding container 7ea4e4cf4a3880d309d16c76630a9a35bada792ac800dc3b80a6eb6405fb8bf6: Status 404 returned error can't find the container with id 7ea4e4cf4a3880d309d16c76630a9a35bada792ac800dc3b80a6eb6405fb8bf6
Oct 30 17:53:16.372621 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.372586    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4760578f-f08d-40d2-b990-efab5dbb4746" path="/var/lib/kubelet/pods/4760578f-f08d-40d2-b990-efab5dbb4746/volumes"
Oct 30 17:53:16.861330 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.861290    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57" event={"ID":"107b59fd-c805-400b-b944-35d563b1020f","Type":"ContainerStarted","Data":"8b0569f64dbde6566082f3908e4131dba69129d35d17307730d74837c4a3ad93"}
Oct 30 17:53:16.861828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.861359    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57" event={"ID":"107b59fd-c805-400b-b944-35d563b1020f","Type":"ContainerStarted","Data":"7ea4e4cf4a3880d309d16c76630a9a35bada792ac800dc3b80a6eb6405fb8bf6"}
Oct 30 17:53:16.862952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.862914    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"]
Oct 30 17:53:16.868857 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.868835    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.888322 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.888297    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c596f1-openshift-default-dockercfg-zrcfh\""
Oct 30 17:53:16.896733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.896709    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"]
Oct 30 17:53:16.903085 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.902596    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57" podStartSLOduration=1.9025800940000002 podStartE2EDuration="1.902580094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:16.901470417 +0000 UTC m=+2402.663514233" watchObservedRunningTime="2026-10-30 17:53:16.902580094 +0000 UTC m=+2402.664623908"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.909918    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.909957    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdx45\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910040    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910149    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910272    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910309    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910461    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910571    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:16.915741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:16.910808    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012019    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012123    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012242    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012279    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012307    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdx45\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012343    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012410    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012508    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012415    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012633    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012699    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.012969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012941    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.013043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.012968    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.013433 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.013377    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.014974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.014953    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.015171 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.015153    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.035511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.035486    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdx45\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.035678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.035604    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.187739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.187659    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:17.232212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.232181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:17.354410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.354305    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:17.481002 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.480920    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:17.530958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.530919    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"]
Oct 30 17:53:17.542236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.542196    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:17.542382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.542285    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:17.542382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.542333    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:17.602105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.599876    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:17.928318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.928284    5534 generic.go:356] "Generic (PLEG): container finished" podID="107b59fd-c805-400b-b944-35d563b1020f" containerID="8b0569f64dbde6566082f3908e4131dba69129d35d17307730d74837c4a3ad93" exitCode=0
Oct 30 17:53:17.929306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.928938    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57" event={"ID":"107b59fd-c805-400b-b944-35d563b1020f","Type":"ContainerDied","Data":"8b0569f64dbde6566082f3908e4131dba69129d35d17307730d74837c4a3ad93"}
Oct 30 17:53:17.931634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.931596    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" event={"ID":"d619970a-5638-4ae2-b0a9-c647e21f496a","Type":"ContainerStarted","Data":"c30c7caeb8f0c695b533ec7d6eba52347bc75216c508ec9a1a50d239acb3d895"}
Oct 30 17:53:17.931773 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:17.931675    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" event={"ID":"d619970a-5638-4ae2-b0a9-c647e21f496a","Type":"ContainerStarted","Data":"8d4bd37b1fa72648b57c7fb1e7d81d8c999877d42128dfe60a813e9983d04823"}
Oct 30 17:53:18.016190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:18.016140    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podStartSLOduration=2.016126148 podStartE2EDuration="2.016126148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:18.012970082 +0000 UTC m=+2403.775013919" watchObservedRunningTime="2026-10-30 17:53:18.016126148 +0000 UTC m=+2403.778169959"
Oct 30 17:53:18.187828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:18.187790    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:18.189806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:18.189778    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:18.189928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:18.189837    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:18.625314 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:18.625276    5534 scope.go:122] "RemoveContainer" containerID="a5f95660777ff99c5b676eb66bb23f2194c54f3f61f1e3ab114a71899c2ffc26"
Oct 30 17:53:18.639072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:18.639049    5534 scope.go:122] "RemoveContainer" containerID="4729bfa9bbbcfd46036b51cec3f750d50aba481fbba3af7d6ede33e91f008e24"
Oct 30 17:53:19.197665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.193074    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:19.197665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.193146    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:19.244798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.244775    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:19.417052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.417018    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca\") pod \"107b59fd-c805-400b-b944-35d563b1020f\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") "
Oct 30 17:53:19.417298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.417279    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca\") pod \"107b59fd-c805-400b-b944-35d563b1020f\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") "
Oct 30 17:53:19.417424 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.417336    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph\" (UniqueName: \"kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph\") pod \"107b59fd-c805-400b-b944-35d563b1020f\" (UID: \"107b59fd-c805-400b-b944-35d563b1020f\") "
Oct 30 17:53:19.417700 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.417674    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca" pod "107b59fd-c805-400b-b944-35d563b1020f" (UID: "107b59fd-c805-400b-b944-35d563b1020f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:19.417837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.417824    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107b59fd-c805-400b-b944-35d563b1020f-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:19.419285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.419262    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca" pod "107b59fd-c805-400b-b944-35d563b1020f" (UID: "107b59fd-c805-400b-b944-35d563b1020f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:53:19.419706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.419684    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph" pod "107b59fd-c805-400b-b944-35d563b1020f" (UID: "107b59fd-c805-400b-b944-35d563b1020f"). InnerVolumeSpecName "kube-api-access-bbfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:19.523724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.522899    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbfph\" (UniqueName: \"kubernetes.io/projected/107b59fd-c805-400b-b944-35d563b1020f-kube-api-access-bbfph\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:19.523724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.522941    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107b59fd-c805-400b-b944-35d563b1020f-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:19.775724 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.770714    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:19.990864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.990731    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"
Oct 30 17:53:19.990864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.990759    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57" event={"ID":"107b59fd-c805-400b-b944-35d563b1020f","Type":"ContainerDied","Data":"7ea4e4cf4a3880d309d16c76630a9a35bada792ac800dc3b80a6eb6405fb8bf6"}
Oct 30 17:53:19.990864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:19.990821    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea4e4cf4a3880d309d16c76630a9a35bada792ac800dc3b80a6eb6405fb8bf6"
Oct 30 17:53:20.189669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.188942    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:20.189669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.189024    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:20.300992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.300941    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:20.891969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.891923    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"]
Oct 30 17:53:20.893163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.893139    5534 cpu_manager.go:414] "containerMap: removing container" podUID="107b59fd-c805-400b-b944-35d563b1020f" containerName="revoke-keys"
Oct 30 17:53:20.893163 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.893162    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107b59fd-c805-400b-b944-35d563b1020f" containerName="revoke-keys"
Oct 30 17:53:20.893397 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.893342    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="107b59fd-c805-400b-b944-35d563b1020f" containerName="revoke-keys"
Oct 30 17:53:20.897897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.897787    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:20.911571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.911550    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"]
Oct 30 17:53:20.917038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.916936    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:53:20.917038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.917008    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:20.997670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.997510    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mlh\" (UniqueName: \"kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:20.997670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.997573    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:20.997966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:20.997931    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.099414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.099375    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mlh\" (UniqueName: \"kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.099633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.099430    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.099769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.099737    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.100466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.100376    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.102261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.102224    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.140216 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.140193    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mlh\" (UniqueName: \"kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh\") pod \"maas-api-revoke-sub-9d72bc1e-n6q2n\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.189733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.188956    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:21.189733 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.189009    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:21.218704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.218635    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:21.594009 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:21.593966    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903d68bb_60cc_49f9_b93b_f452378a86dd.slice/crio-91863d40e7141d94b49dcb32d327f1480190bf2e902717fed1235404e74d8982 WatchSource:0}: Error finding container 91863d40e7141d94b49dcb32d327f1480190bf2e902717fed1235404e74d8982: Status 404 returned error can't find the container with id 91863d40e7141d94b49dcb32d327f1480190bf2e902717fed1235404e74d8982
Oct 30 17:53:21.599353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:21.598544    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"]
Oct 30 17:53:22.016839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:22.016796    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n" event={"ID":"903d68bb-60cc-49f9-b93b-f452378a86dd","Type":"ContainerStarted","Data":"91863d40e7141d94b49dcb32d327f1480190bf2e902717fed1235404e74d8982"}
Oct 30 17:53:22.187274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:22.187015    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:22.188599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:22.188561    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:22.188837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:22.188798    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:23.026175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:23.026139    5534 generic.go:356] "Generic (PLEG): container finished" podID="903d68bb-60cc-49f9-b93b-f452378a86dd" containerID="4bcc742484f2f3ca99ac1b8b1316e45f584f332798b36c7bd7ad49ab5bdf0571" exitCode=0
Oct 30 17:53:23.026638 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:23.026237    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n" event={"ID":"903d68bb-60cc-49f9-b93b-f452378a86dd","Type":"ContainerDied","Data":"4bcc742484f2f3ca99ac1b8b1316e45f584f332798b36c7bd7ad49ab5bdf0571"}
Oct 30 17:53:23.189125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:23.189079    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:23.189689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:23.189144    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:23.890903 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:23.890862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:24.188991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.188958    5534 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 17:53:24.189428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.189023    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 17:53:24.216532 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.216510    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:24.248895 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.248854    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca\") pod \"903d68bb-60cc-49f9-b93b-f452378a86dd\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") "
Oct 30 17:53:24.249075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.249059    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh\" (UniqueName: \"kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh\") pod \"903d68bb-60cc-49f9-b93b-f452378a86dd\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") "
Oct 30 17:53:24.249134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.249103    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca\") pod \"903d68bb-60cc-49f9-b93b-f452378a86dd\" (UID: \"903d68bb-60cc-49f9-b93b-f452378a86dd\") "
Oct 30 17:53:24.249630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.249531    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca" pod "903d68bb-60cc-49f9-b93b-f452378a86dd" (UID: "903d68bb-60cc-49f9-b93b-f452378a86dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:24.250273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.250248    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:24.251959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.251933    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca" pod "903d68bb-60cc-49f9-b93b-f452378a86dd" (UID: "903d68bb-60cc-49f9-b93b-f452378a86dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:53:24.252073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.252054    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh" pod "903d68bb-60cc-49f9-b93b-f452378a86dd" (UID: "903d68bb-60cc-49f9-b93b-f452378a86dd"). InnerVolumeSpecName "kube-api-access-l7mlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:24.351503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.351469    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/903d68bb-60cc-49f9-b93b-f452378a86dd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:24.351503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:24.351504    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mlh\" (UniqueName: \"kubernetes.io/projected/903d68bb-60cc-49f9-b93b-f452378a86dd-kube-api-access-l7mlh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:25.045943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.045912    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"
Oct 30 17:53:25.046125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.045957    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n" event={"ID":"903d68bb-60cc-49f9-b93b-f452378a86dd","Type":"ContainerDied","Data":"91863d40e7141d94b49dcb32d327f1480190bf2e902717fed1235404e74d8982"}
Oct 30 17:53:25.046125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.046024    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91863d40e7141d94b49dcb32d327f1480190bf2e902717fed1235404e74d8982"
Oct 30 17:53:25.193742 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.193702    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:25.194234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.194154    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:25.195049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.195030    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:53:25.595661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.595602    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:25.946431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:25.946354    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:27.402078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:27.402044    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:29.063528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:29.063493    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:30.999902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:30.999869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:32.386068 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.386038    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:32.530675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.526711    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:32.708307 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.708226    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"]
Oct 30 17:53:32.708547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.708522    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" containerID="cri-o://382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e" gracePeriod=30
Oct 30 17:53:32.721093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.721054    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"]
Oct 30 17:53:32.729890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.729858    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:32.879276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:32.879227    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:33.100850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.100820    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:33.119794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.119757    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" containerID="cri-o://71fc1b9c2dd52dbbf5337ed16c73eb7436e3b53b16877e8e0f92d985b80b696a" gracePeriod=30
Oct 30 17:53:33.388348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.388241    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"]
Oct 30 17:53:33.389079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.389061    5534 cpu_manager.go:414] "containerMap: removing container" podUID="903d68bb-60cc-49f9-b93b-f452378a86dd" containerName="revoke-keys"
Oct 30 17:53:33.389079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.389079    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903d68bb-60cc-49f9-b93b-f452378a86dd" containerName="revoke-keys"
Oct 30 17:53:33.389233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.389223    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="903d68bb-60cc-49f9-b93b-f452378a86dd" containerName="revoke-keys"
Oct 30 17:53:33.393137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.393117    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.415041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.415012    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"]
Oct 30 17:53:33.475222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.475188    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:33.475432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.475394    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:53:33.510293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.510258    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.510499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.510422    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.510599 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.510585    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhtf2\" (UniqueName: \"kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.611696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.611622    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.611896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.611770    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.611951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.611924    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhtf2\" (UniqueName: \"kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.612428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.612404    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.614561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.614540    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.642589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.642524    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhtf2\" (UniqueName: \"kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2\") pod \"maas-api-revoke-keys-e2e-inf-a-043253-bpzmq\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:33.708091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:33.708056    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:34.039997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:34.039964    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"]
Oct 30 17:53:34.040418 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:34.040394    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6922c7f_5c7a_410c_b891_e7cd46a0df17.slice/crio-898ccd293c2f9c4c2669f917490fdba3729fdc90259c41be29c0e38ff58414f0 WatchSource:0}: Error finding container 898ccd293c2f9c4c2669f917490fdba3729fdc90259c41be29c0e38ff58414f0: Status 404 returned error can't find the container with id 898ccd293c2f9c4c2669f917490fdba3729fdc90259c41be29c0e38ff58414f0
Oct 30 17:53:34.129531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:34.129498    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq" event={"ID":"c6922c7f-5c7a-410c-b891-e7cd46a0df17","Type":"ContainerStarted","Data":"898ccd293c2f9c4c2669f917490fdba3729fdc90259c41be29c0e38ff58414f0"}
Oct 30 17:53:34.900331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:34.900296    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:35.141634 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:35.141595    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq" event={"ID":"c6922c7f-5c7a-410c-b891-e7cd46a0df17","Type":"ContainerStarted","Data":"f0a5bb4a1e65ecf5aaf5f86e2bd77085af520844df1f6d8e3dfaed9b85ba6671"}
Oct 30 17:53:36.153519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:36.153487    5534 generic.go:356] "Generic (PLEG): container finished" podID="c6922c7f-5c7a-410c-b891-e7cd46a0df17" containerID="f0a5bb4a1e65ecf5aaf5f86e2bd77085af520844df1f6d8e3dfaed9b85ba6671" exitCode=0
Oct 30 17:53:36.154001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:36.153580    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq" event={"ID":"c6922c7f-5c7a-410c-b891-e7cd46a0df17","Type":"ContainerDied","Data":"f0a5bb4a1e65ecf5aaf5f86e2bd77085af520844df1f6d8e3dfaed9b85ba6671"}
Oct 30 17:53:36.928980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:36.924466    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:53:36.928980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:36.924536    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:53:37.159677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.159556    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:37.523038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.522745    5534 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:53:37.523038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.522813    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:53:37.686143 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.686117    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:37.785064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.785031    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca\") pod \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") "
Oct 30 17:53:37.785361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.785341    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2\" (UniqueName: \"kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2\") pod \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") "
Oct 30 17:53:37.785493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.785409    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca\") pod \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\" (UID: \"c6922c7f-5c7a-410c-b891-e7cd46a0df17\") "
Oct 30 17:53:37.786003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.785958    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca" pod "c6922c7f-5c7a-410c-b891-e7cd46a0df17" (UID: "c6922c7f-5c7a-410c-b891-e7cd46a0df17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:37.789060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.789020    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca" pod "c6922c7f-5c7a-410c-b891-e7cd46a0df17" (UID: "c6922c7f-5c7a-410c-b891-e7cd46a0df17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:53:37.797391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.796822    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2" pod "c6922c7f-5c7a-410c-b891-e7cd46a0df17" (UID: "c6922c7f-5c7a-410c-b891-e7cd46a0df17"). InnerVolumeSpecName "kube-api-access-rhtf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:37.895381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.895348    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:37.895381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.895380    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhtf2\" (UniqueName: \"kubernetes.io/projected/c6922c7f-5c7a-410c-b891-e7cd46a0df17-kube-api-access-rhtf2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:37.899338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.895394    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6922c7f-5c7a-410c-b891-e7cd46a0df17-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:37.993080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:37.993044    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:53:38.098592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.098558    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.098932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.098907    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.099113 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099096    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.099287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099273    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.099404 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099390    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.099526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099509    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.099701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099684    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.099856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099838    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.100008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.099994    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs\") pod \"050346bb-c68b-4503-a695-185ed33cbf6c\" (UID: \"050346bb-c68b-4503-a695-185ed33cbf6c\") "
Oct 30 17:53:38.101422 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.101400    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.102019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.101990    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.102179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.102150    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:38.102552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.102515    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.102792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.102770    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.106547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.106451    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:38.106547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.106473    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "kube-api-access-xshkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:38.106547 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.106512    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:53:38.106957 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.106938    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy" pod "050346bb-c68b-4503-a695-185ed33cbf6c" (UID: "050346bb-c68b-4503-a695-185ed33cbf6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.202284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202255    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202284    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202293    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshkb\" (UniqueName: \"kubernetes.io/projected/050346bb-c68b-4503-a695-185ed33cbf6c-kube-api-access-xshkb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202303    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202311    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202319    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050346bb-c68b-4503-a695-185ed33cbf6c-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202327    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050346bb-c68b-4503-a695-185ed33cbf6c-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202336    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.202740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.202344    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050346bb-c68b-4503-a695-185ed33cbf6c-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.258946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.258916    5534 generic.go:356] "Generic (PLEG): container finished" podID="050346bb-c68b-4503-a695-185ed33cbf6c" containerID="382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e" exitCode=0
Oct 30 17:53:38.259151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.259014    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" event={"ID":"050346bb-c68b-4503-a695-185ed33cbf6c","Type":"ContainerDied","Data":"382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e"}
Oct 30 17:53:38.259151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.259065    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j" event={"ID":"050346bb-c68b-4503-a695-185ed33cbf6c","Type":"ContainerDied","Data":"c7c5b3e433906bc9017dfaf9f06b52feafb650e0949d6ba78efc09bf626a1462"}
Oct 30 17:53:38.259151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.259095    5534 scope.go:122] "RemoveContainer" containerID="382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e"
Oct 30 17:53:38.259355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.259248    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"
Oct 30 17:53:38.267979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.265746    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq" event={"ID":"c6922c7f-5c7a-410c-b891-e7cd46a0df17","Type":"ContainerDied","Data":"898ccd293c2f9c4c2669f917490fdba3729fdc90259c41be29c0e38ff58414f0"}
Oct 30 17:53:38.267979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.265796    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898ccd293c2f9c4c2669f917490fdba3729fdc90259c41be29c0e38ff58414f0"
Oct 30 17:53:38.267979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.265856    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"
Oct 30 17:53:38.269919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.269194    5534 generic.go:356] "Generic (PLEG): container finished" podID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerID="71fc1b9c2dd52dbbf5337ed16c73eb7436e3b53b16877e8e0f92d985b80b696a" exitCode=0
Oct 30 17:53:38.269919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.269315    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" event={"ID":"122c81ef-c322-48bb-9a02-45fd6359b6cd","Type":"ContainerDied","Data":"71fc1b9c2dd52dbbf5337ed16c73eb7436e3b53b16877e8e0f92d985b80b696a"}
Oct 30 17:53:38.281889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.281812    5534 scope.go:122] "RemoveContainer" containerID="5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"
Oct 30 17:53:38.296290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.296227    5534 scope.go:122] "RemoveContainer" containerID="382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e"
Oct 30 17:53:38.296625 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:53:38.296597    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e\": container with ID starting with 382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e not found: ID does not exist" containerID="382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e"
Oct 30 17:53:38.296836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.296705    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e"} err="failed to get container status \"382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e\": rpc error: code = NotFound desc = could not find container \"382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e\": container with ID starting with 382c8936a93c32db53ce61570a0471c6ee564835a69d7a4401c47c723e9bb51e not found: ID does not exist"
Oct 30 17:53:38.296836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.296734    5534 scope.go:122] "RemoveContainer" containerID="5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"
Oct 30 17:53:38.297230 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:53:38.297086    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571\": container with ID starting with 5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571 not found: ID does not exist" containerID="5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"
Oct 30 17:53:38.297230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.297116    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571"} err="failed to get container status \"5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571\": rpc error: code = NotFound desc = could not find container \"5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571\": container with ID starting with 5d043d77347c20f0a1e856365d85fdcf2cedbc319da6d9ec38f9dd91b1ef4571 not found: ID does not exist"
Oct 30 17:53:38.328685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.324428    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"]
Oct 30 17:53:38.330855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.330829    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-lmr4j"]
Oct 30 17:53:38.398591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.398493    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" path="/var/lib/kubelet/pods/050346bb-c68b-4503-a695-185ed33cbf6c/volumes"
Oct 30 17:53:38.413947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.413924    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:53:38.507873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.507836    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.507920    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.507989    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.508153    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508237 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.508189    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.508263    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.508360    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.508410    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.508484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.508464    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy\") pod \"122c81ef-c322-48bb-9a02-45fd6359b6cd\" (UID: \"122c81ef-c322-48bb-9a02-45fd6359b6cd\") "
Oct 30 17:53:38.510706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.510134    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:38.512430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.512012    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.512430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.512215    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.513999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.512713    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.515141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.514301    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.516347 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.516318    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:38.527815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.527789    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:53:38.527965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.527884    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:53:38.529074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.529054    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l" pod "122c81ef-c322-48bb-9a02-45fd6359b6cd" (UID: "122c81ef-c322-48bb-9a02-45fd6359b6cd"). InnerVolumeSpecName "kube-api-access-68n8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:38.611688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611632    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122c81ef-c322-48bb-9a02-45fd6359b6cd-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611688    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611701    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611713    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611724    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611735    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611746    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122c81ef-c322-48bb-9a02-45fd6359b6cd-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611757    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122c81ef-c322-48bb-9a02-45fd6359b6cd-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.611911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:38.611769    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68n8l\" (UniqueName: \"kubernetes.io/projected/122c81ef-c322-48bb-9a02-45fd6359b6cd-kube-api-access-68n8l\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:39.281526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:39.281480    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"
Oct 30 17:53:39.281526 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:39.281516    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp" event={"ID":"122c81ef-c322-48bb-9a02-45fd6359b6cd","Type":"ContainerDied","Data":"49484e620ed4fadbc72e60ddc96b166fb327e0efc59ab1943d0ffde4193a202d"}
Oct 30 17:53:39.282126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:39.281583    5534 scope.go:122] "RemoveContainer" containerID="71fc1b9c2dd52dbbf5337ed16c73eb7436e3b53b16877e8e0f92d985b80b696a"
Oct 30 17:53:39.347250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:39.347219    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"]
Oct 30 17:53:39.359452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:39.359422    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-qtwhp"]
Oct 30 17:53:39.396086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:39.396060    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:40.371884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:40.371848    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" path="/var/lib/kubelet/pods/122c81ef-c322-48bb-9a02-45fd6359b6cd/volumes"
Oct 30 17:53:41.018219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:41.018186    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:42.848164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:42.848133    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:42.961342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:42.961304    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:43.072052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:43.071743    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:43.817447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:43.817411    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"]
Oct 30 17:53:43.857132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:43.857101    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"]
Oct 30 17:53:43.968233 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:43.968194    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:44.087737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:44.083024    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:44.668674 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:44.668623    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:45.253122 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:53:45.253076    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:53:46.405619 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:46.405585    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:47.680555 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:47.680512    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"]
Oct 30 17:53:47.727495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:47.727455    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:48.198726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.198696    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:48.293552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.293518    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"]
Oct 30 17:53:48.294274 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294259    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c6922c7f-5c7a-410c-b891-e7cd46a0df17" containerName="revoke-keys"
Oct 30 17:53:48.294327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294277    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6922c7f-5c7a-410c-b891-e7cd46a0df17" containerName="revoke-keys"
Oct 30 17:53:48.294327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294307    5534 cpu_manager.go:414] "containerMap: removing container" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy"
Oct 30 17:53:48.294327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294314    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy"
Oct 30 17:53:48.294421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294349    5534 cpu_manager.go:414] "containerMap: removing container" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy"
Oct 30 17:53:48.294421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294357    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy"
Oct 30 17:53:48.294421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294372    5534 cpu_manager.go:414] "containerMap: removing container" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy"
Oct 30 17:53:48.294421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294377    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy"
Oct 30 17:53:48.294572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294489    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy"
Oct 30 17:53:48.294572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294498    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="050346bb-c68b-4503-a695-185ed33cbf6c" containerName="istio-proxy"
Oct 30 17:53:48.294572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294507    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6922c7f-5c7a-410c-b891-e7cd46a0df17" containerName="revoke-keys"
Oct 30 17:53:48.294572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.294517    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="122c81ef-c322-48bb-9a02-45fd6359b6cd" containerName="istio-proxy"
Oct 30 17:53:48.299580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.299562    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.314412 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.314343    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:48.317763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.317737    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"]
Oct 30 17:53:48.324956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.324933    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:53:48.374872 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.374838    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwqz\" (UniqueName: \"kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.375522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.375497    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.375831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.375797    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.470084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.469962    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:48.477259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.477225    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwqz\" (UniqueName: \"kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.477401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.477362    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.477495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.477464    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.478264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.478240    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.479959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.479936    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.501668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.500766    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwqz\" (UniqueName: \"kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz\") pod \"maas-api-revoke-sub-53e6a9bc-xnrb2\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.614200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.614157    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:48.832698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.825457    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm" podUID="6b07c046-bdbd-451c-83d6-1b8e50b1b010" containerName="payload-processing" containerID="cri-o://289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c" gracePeriod=25
Oct 30 17:53:48.860373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.859159    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerName="payload-pre-processing" containerID="cri-o://9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5" gracePeriod=25
Oct 30 17:53:49.007317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:48.995343    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"]
Oct 30 17:53:49.180703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.180676    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:53:49.208882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.208853    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume\") pod \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") "
Oct 30 17:53:49.209174 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.209156    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6\" (UniqueName: \"kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6\") pod \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\" (UID: \"6b07c046-bdbd-451c-83d6-1b8e50b1b010\") "
Oct 30 17:53:49.209243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.209209    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume" pod "6b07c046-bdbd-451c-83d6-1b8e50b1b010" (UID: "6b07c046-bdbd-451c-83d6-1b8e50b1b010"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:49.209761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.209703    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b07c046-bdbd-451c-83d6-1b8e50b1b010-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:49.211372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.211347    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6" pod "6b07c046-bdbd-451c-83d6-1b8e50b1b010" (UID: "6b07c046-bdbd-451c-83d6-1b8e50b1b010"). InnerVolumeSpecName "kube-api-access-p27v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:49.311029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.310946    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p27v6\" (UniqueName: \"kubernetes.io/projected/6b07c046-bdbd-451c-83d6-1b8e50b1b010-kube-api-access-p27v6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:49.384877 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.384843    5534 generic.go:356] "Generic (PLEG): container finished" podID="6b07c046-bdbd-451c-83d6-1b8e50b1b010" containerID="289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c" exitCode=0
Oct 30 17:53:49.385024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.384894    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"
Oct 30 17:53:49.385024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.384924    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm" event={"ID":"6b07c046-bdbd-451c-83d6-1b8e50b1b010","Type":"ContainerDied","Data":"289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c"}
Oct 30 17:53:49.385024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.384980    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm" event={"ID":"6b07c046-bdbd-451c-83d6-1b8e50b1b010","Type":"ContainerDied","Data":"3252dbba3718cdcb9002600bf02e64126152fc0e3fa38e4e170b21d6634db8d5"}
Oct 30 17:53:49.385024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.385008    5534 scope.go:122] "RemoveContainer" containerID="289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c"
Oct 30 17:53:49.386946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.386921    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2" event={"ID":"8d1f8e56-5273-4792-af5c-38c27c304ccd","Type":"ContainerStarted","Data":"4e70a04b647f8e95f1e3037ca3ffde9033a13031dd5db1cc811c1b5b9b701bc0"}
Oct 30 17:53:49.387067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.386969    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2" event={"ID":"8d1f8e56-5273-4792-af5c-38c27c304ccd","Type":"ContainerStarted","Data":"a690e2dd82a9813c22288b482f8ba59b58a600bf0856c4b24a9a13571fc04fb4"}
Oct 30 17:53:49.397600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.397582    5534 scope.go:122] "RemoveContainer" containerID="289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c"
Oct 30 17:53:49.397965 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:53:49.397941    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c\": container with ID starting with 289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c not found: ID does not exist" containerID="289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c"
Oct 30 17:53:49.398053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.397973    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c"} err="failed to get container status \"289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c\": rpc error: code = NotFound desc = could not find container \"289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c\": container with ID starting with 289a691f10b4b37ff38617215dcb85acfedaee4bd4c6725ebd0ac8ea745f548c not found: ID does not exist"
Oct 30 17:53:49.439017 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.438971    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"]
Oct 30 17:53:49.442776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.442751    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-043253-8587649dc4-49lwm"]
Oct 30 17:53:49.477996 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:49.477948    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2" podStartSLOduration=1.477933582 podStartE2EDuration="1.477933582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:49.47521706 +0000 UTC m=+2435.237260874" watchObservedRunningTime="2026-10-30 17:53:49.477933582 +0000 UTC m=+2435.239977392"
Oct 30 17:53:50.046482 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:50.046446    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:50.375792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:50.375756    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b07c046-bdbd-451c-83d6-1b8e50b1b010" path="/var/lib/kubelet/pods/6b07c046-bdbd-451c-83d6-1b8e50b1b010/volumes"
Oct 30 17:53:50.398966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:50.398933    5534 generic.go:356] "Generic (PLEG): container finished" podID="8d1f8e56-5273-4792-af5c-38c27c304ccd" containerID="4e70a04b647f8e95f1e3037ca3ffde9033a13031dd5db1cc811c1b5b9b701bc0" exitCode=0
Oct 30 17:53:50.399130 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:50.399018    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2" event={"ID":"8d1f8e56-5273-4792-af5c-38c27c304ccd","Type":"ContainerDied","Data":"4e70a04b647f8e95f1e3037ca3ffde9033a13031dd5db1cc811c1b5b9b701bc0"}
Oct 30 17:53:51.585047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.585018    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:51.646332 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.646299    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz\" (UniqueName: \"kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz\") pod \"8d1f8e56-5273-4792-af5c-38c27c304ccd\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") "
Oct 30 17:53:51.646528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.646400    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca\") pod \"8d1f8e56-5273-4792-af5c-38c27c304ccd\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") "
Oct 30 17:53:51.646528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.646444    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca\") pod \"8d1f8e56-5273-4792-af5c-38c27c304ccd\" (UID: \"8d1f8e56-5273-4792-af5c-38c27c304ccd\") "
Oct 30 17:53:51.646887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.646854    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca" pod "8d1f8e56-5273-4792-af5c-38c27c304ccd" (UID: "8d1f8e56-5273-4792-af5c-38c27c304ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:51.647126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.647104    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:51.649808 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.649782    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz" pod "8d1f8e56-5273-4792-af5c-38c27c304ccd" (UID: "8d1f8e56-5273-4792-af5c-38c27c304ccd"). InnerVolumeSpecName "kube-api-access-2mwqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:51.650010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.649991    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca" pod "8d1f8e56-5273-4792-af5c-38c27c304ccd" (UID: "8d1f8e56-5273-4792-af5c-38c27c304ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:53:51.748902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.748868    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f8e56-5273-4792-af5c-38c27c304ccd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:51.748902 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.748900    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwqz\" (UniqueName: \"kubernetes.io/projected/8d1f8e56-5273-4792-af5c-38c27c304ccd-kube-api-access-2mwqz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:53:51.779967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:51.779930    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:52.418298 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:52.418271    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"
Oct 30 17:53:52.418476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:52.418310    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2" event={"ID":"8d1f8e56-5273-4792-af5c-38c27c304ccd","Type":"ContainerDied","Data":"a690e2dd82a9813c22288b482f8ba59b58a600bf0856c4b24a9a13571fc04fb4"}
Oct 30 17:53:52.418476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:52.418362    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a690e2dd82a9813c22288b482f8ba59b58a600bf0856c4b24a9a13571fc04fb4"
Oct 30 17:53:53.368438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:53.368387    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:54.993256 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:54.993173    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:55.137812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:55.137772    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:53:56.772073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772036    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"]
Oct 30 17:53:56.772843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772824    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8d1f8e56-5273-4792-af5c-38c27c304ccd" containerName="revoke-keys"
Oct 30 17:53:56.772843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772842    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1f8e56-5273-4792-af5c-38c27c304ccd" containerName="revoke-keys"
Oct 30 17:53:56.773020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772854    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6b07c046-bdbd-451c-83d6-1b8e50b1b010" containerName="payload-processing"
Oct 30 17:53:56.773020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772859    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b07c046-bdbd-451c-83d6-1b8e50b1b010" containerName="payload-processing"
Oct 30 17:53:56.773020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772973    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1f8e56-5273-4792-af5c-38c27c304ccd" containerName="revoke-keys"
Oct 30 17:53:56.773020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.772982    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b07c046-bdbd-451c-83d6-1b8e50b1b010" containerName="payload-processing"
Oct 30 17:53:56.777142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.777120    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.793610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.793392    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a23c98-serving-cert\""
Oct 30 17:53:56.800162 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.800055    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"]
Oct 30 17:53:56.829815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.829786    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.830013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.829988    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.830097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.830081    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.830202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.830188    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz87n\" (UniqueName: \"kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.848411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.848377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:56.931676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.931623    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.931835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.931692    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.931835 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.931809    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz87n\" (UniqueName: \"kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.931933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.931844    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.932510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.932483    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.934390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.934367    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.934513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.934482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.956564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.956536    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz87n\" (UniqueName: \"kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n\") pod \"maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:56.963419 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:56.963389    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:57.098942 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:57.098910    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:57.419792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:57.419762    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"]
Oct 30 17:53:57.420549 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:57.420521    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6462b7fa_fcbd_4c93_a2b7_45fc9b722fdd.slice/crio-9a9a14595f91d051c3bf72ec391fd9ddd5d2ebf242808124ba55fc25665fe6da WatchSource:0}: Error finding container 9a9a14595f91d051c3bf72ec391fd9ddd5d2ebf242808124ba55fc25665fe6da: Status 404 returned error can't find the container with id 9a9a14595f91d051c3bf72ec391fd9ddd5d2ebf242808124ba55fc25665fe6da
Oct 30 17:53:57.422092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:57.422074    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:53:57.466518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:57.466487    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7" event={"ID":"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd","Type":"ContainerStarted","Data":"9a9a14595f91d051c3bf72ec391fd9ddd5d2ebf242808124ba55fc25665fe6da"}
Oct 30 17:53:58.040987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.040934    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"]
Oct 30 17:53:58.094682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.094628    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc38ecd5-z7qtv"]
Oct 30 17:53:58.374366 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.374287    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f1ba1c-1014-4c63-bbe4-eabb980fc254" path="/var/lib/kubelet/pods/83f1ba1c-1014-4c63-bbe4-eabb980fc254/volumes"
Oct 30 17:53:58.492712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.492631    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7" event={"ID":"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd","Type":"ContainerStarted","Data":"44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0"}
Oct 30 17:53:58.492923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.492866    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:53:58.537272 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.537220    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7" podStartSLOduration=2.134857042 podStartE2EDuration="2.537206844s" totalImagesPullingTime="402.349802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:58.534132221 +0000 UTC m=+2444.296176033" watchObservedRunningTime="2026-10-30 17:53:58.537206844 +0000 UTC m=+2444.299250692"
Oct 30 17:53:58.628193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.628108    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:53:58.857153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.857114    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"]
Oct 30 17:53:58.862061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.862039    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.876850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.876828    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-c31c73-openshift-default-dockercfg-qnblk\""
Oct 30 17:53:58.897175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.897100    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"]
Oct 30 17:53:58.973031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.972980    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqvz\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973037    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973222    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973280    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973381    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973423    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973531    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973555    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:58.973602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:58.973581    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.030906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.030874    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"]
Oct 30 17:53:59.034605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.034583    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e46ca3-lftrt"]
Oct 30 17:53:59.074557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074524    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074576    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074625    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074674    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074731    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074750    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074782    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074830    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqvz\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.074866    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.075198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.075232    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.075410    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075570 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.075452    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.075837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.075812    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.077592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.077569    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.077852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.077834    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.097911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.097889    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqvz\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.098035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.097930    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.178912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.178811    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:53:59.506550 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:53:59.506516    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6a8730_f4fa_4440_8994_5cad740314f1.slice/crio-33dee7a610d7068aec808c85ab2ef4a9035bc85d0241b5dc4b045564eb982f3b WatchSource:0}: Error finding container 33dee7a610d7068aec808c85ab2ef4a9035bc85d0241b5dc4b045564eb982f3b: Status 404 returned error can't find the container with id 33dee7a610d7068aec808c85ab2ef4a9035bc85d0241b5dc4b045564eb982f3b
Oct 30 17:53:59.507884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.507860    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"]
Oct 30 17:53:59.509380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.509349    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:59.509472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.509415    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:53:59.509472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:53:59.509464    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:54:00.404099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:00.404061    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff00a7e-4ba0-4110-8d54-fcb543ec6495" path="/var/lib/kubelet/pods/fff00a7e-4ba0-4110-8d54-fcb543ec6495/volumes"
Oct 30 17:54:00.513282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:00.513235    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" event={"ID":"0c6a8730-f4fa-4440-8994-5cad740314f1","Type":"ContainerStarted","Data":"3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a"}
Oct 30 17:54:00.513481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:00.513309    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" event={"ID":"0c6a8730-f4fa-4440-8994-5cad740314f1","Type":"ContainerStarted","Data":"33dee7a610d7068aec808c85ab2ef4a9035bc85d0241b5dc4b045564eb982f3b"}
Oct 30 17:54:00.515000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:00.514978    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:00.582668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:00.579214    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podStartSLOduration=2.579195789 podStartE2EDuration="2.579195789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:00.578040656 +0000 UTC m=+2446.340084470" watchObservedRunningTime="2026-10-30 17:54:00.579195789 +0000 UTC m=+2446.341239604"
Oct 30 17:54:01.182675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:01.182003    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:54:01.186576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:01.186092    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 17:54:01.186576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:01.186162    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 17:54:02.181047 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:02.181007    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 17:54:02.181550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:02.181072    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 17:54:02.577725 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:02.577146    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:02.741281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:02.741133    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" podUID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" containerName="main" containerID="cri-o://a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968" gracePeriod=45
Oct 30 17:54:02.784668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:02.781517    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" podUID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.237:8000/ready\": dial tcp 10.133.0.237:8000: connect: connection refused"
Oct 30 17:54:02.980244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:02.979926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:03.217361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.193115    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 17:54:03.217361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.193181    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 17:54:03.440393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.440357    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:54:03.612671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.610574    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs\") pod \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") "
Oct 30 17:54:03.612671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.610671    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache\") pod \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") "
Oct 30 17:54:03.612671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.610864    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm\") pod \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") "
Oct 30 17:54:03.612671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.610995    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home\") pod \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") "
Oct 30 17:54:03.612671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.611024    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir\") pod \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") "
Oct 30 17:54:03.612671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.611060    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6\" (UniqueName: \"kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6\") pod \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\" (UID: \"473abcc0-5027-4e5a-8c7e-d8f46645f1ad\") "
Oct 30 17:54:03.651702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.643610    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6" pod "473abcc0-5027-4e5a-8c7e-d8f46645f1ad" (UID: "473abcc0-5027-4e5a-8c7e-d8f46645f1ad"). InnerVolumeSpecName "kube-api-access-k6dz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:03.651702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.644315    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache" pod "473abcc0-5027-4e5a-8c7e-d8f46645f1ad" (UID: "473abcc0-5027-4e5a-8c7e-d8f46645f1ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:03.651702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.645009    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home" pod "473abcc0-5027-4e5a-8c7e-d8f46645f1ad" (UID: "473abcc0-5027-4e5a-8c7e-d8f46645f1ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:03.651702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.646365    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir" pod "473abcc0-5027-4e5a-8c7e-d8f46645f1ad" (UID: "473abcc0-5027-4e5a-8c7e-d8f46645f1ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:03.761688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.746251    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:03.761688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.746293    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:03.761688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.746307    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:03.761688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.746322    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6dz6\" (UniqueName: \"kubernetes.io/projected/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-kube-api-access-k6dz6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:03.785837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.780327    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs" pod "473abcc0-5027-4e5a-8c7e-d8f46645f1ad" (UID: "473abcc0-5027-4e5a-8c7e-d8f46645f1ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:54:03.808834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.808724    5534 generic.go:356] "Generic (PLEG): container finished" podID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" containerID="a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968" exitCode=0
Oct 30 17:54:03.808834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.808801    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" event={"ID":"473abcc0-5027-4e5a-8c7e-d8f46645f1ad","Type":"ContainerDied","Data":"a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968"}
Oct 30 17:54:03.809199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.809157    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx" event={"ID":"473abcc0-5027-4e5a-8c7e-d8f46645f1ad","Type":"ContainerDied","Data":"5e8f626033733b19a44562e926ee491e453eb7349b3fbd4c3c233e00b1f7c98d"}
Oct 30 17:54:03.809372 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.809308    5534 scope.go:122] "RemoveContainer" containerID="a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968"
Oct 30 17:54:03.810188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:03.809702    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"
Oct 30 17:54:04.087577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.087544    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:04.146704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.144961    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm" pod "473abcc0-5027-4e5a-8c7e-d8f46645f1ad" (UID: "473abcc0-5027-4e5a-8c7e-d8f46645f1ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:04.221675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.215710    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 17:54:04.221675 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.215805    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 17:54:04.222201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.221856    5534 scope.go:122] "RemoveContainer" containerID="a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968"
Oct 30 17:54:04.222328 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:04.222292    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968\": container with ID starting with a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968 not found: ID does not exist" containerID="a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968"
Oct 30 17:54:04.222413 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.222333    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968"} err="failed to get container status \"a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968\": rpc error: code = NotFound desc = could not find container \"a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968\": container with ID starting with a612be4660f9ddb37b2cf361fa2ff2972eb206c461a4870817a9110765c7a968 not found: ID does not exist"
Oct 30 17:54:04.248473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.248432    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473abcc0-5027-4e5a-8c7e-d8f46645f1ad-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:04.516691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.516633    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 17:54:04.547538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.546772    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"]
Oct 30 17:54:04.671665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:04.656750    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4a271c/test-model-4a271c-kserve-6c8f689955-nlfgx"]
Oct 30 17:54:05.035508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:05.035476    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:05.154811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:05.154772    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:54:05.185152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:05.185101    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 17:54:05.185471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:05.185439    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 17:54:06.179333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:06.179295    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 17:54:06.179918 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:06.179365    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 17:54:06.382091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:06.382056    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" path="/var/lib/kubelet/pods/473abcc0-5027-4e5a-8c7e-d8f46645f1ad/volumes"
Oct 30 17:54:06.893750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:06.893715    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:07.187798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:07.187715    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:54:07.188705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:07.188675    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:54:07.189600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:07.189577    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 17:54:08.656856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:08.656818    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:09.032217 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.032187    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"]
Oct 30 17:54:09.035775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.035752    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7047efaf-b8lsn"]
Oct 30 17:54:09.160790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.160757    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:54:09.205874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.205840    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume\") pod \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") "
Oct 30 17:54:09.206059 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.205993    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q\" (UniqueName: \"kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q\") pod \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\" (UID: \"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8\") "
Oct 30 17:54:09.206251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.206226    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume" pod "0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" (UID: "0ac24c3e-c7d1-495a-8b4d-8a87d28087a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:09.206814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.206777    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:09.209032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.208809    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q" pod "0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" (UID: "0ac24c3e-c7d1-495a-8b4d-8a87d28087a8"). InnerVolumeSpecName "kube-api-access-4rb6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:09.308235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.308202    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rb6q\" (UniqueName: \"kubernetes.io/projected/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8-kube-api-access-4rb6q\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:09.932540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.932507    5534 generic.go:356] "Generic (PLEG): container finished" podID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerID="9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5" exitCode=0
Oct 30 17:54:09.933019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.932549    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"
Oct 30 17:54:09.933019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.932594    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" event={"ID":"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8","Type":"ContainerDied","Data":"9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5"}
Oct 30 17:54:09.933019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.932664    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x" event={"ID":"0ac24c3e-c7d1-495a-8b4d-8a87d28087a8","Type":"ContainerDied","Data":"3b65ec07f0803175cfae4519d308c1550b63a8a7d910510301485a46eb373c1f"}
Oct 30 17:54:09.933019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.932693    5534 scope.go:122] "RemoveContainer" containerID="9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5"
Oct 30 17:54:09.946600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.946533    5534 scope.go:122] "RemoveContainer" containerID="9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5"
Oct 30 17:54:09.946982 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:09.946962    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5\": container with ID starting with 9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5 not found: ID does not exist" containerID="9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5"
Oct 30 17:54:09.947074 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.946990    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5"} err="failed to get container status \"9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5\": rpc error: code = NotFound desc = could not find container \"9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5\": container with ID starting with 9a76050db6eae1192a18ff52509326f5d946fbc58d8d606d1b832c6dae940da5 not found: ID does not exist"
Oct 30 17:54:09.985369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.985337    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"]
Oct 30 17:54:09.997014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:09.996987    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-043253-5775bc4f6c-5677x"]
Oct 30 17:54:10.373540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:10.373504    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" path="/var/lib/kubelet/pods/0ac24c3e-c7d1-495a-8b4d-8a87d28087a8/volumes"
Oct 30 17:54:10.374123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:10.374099    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad01cb1-91b5-4975-aa1f-52e582e75e06" path="/var/lib/kubelet/pods/2ad01cb1-91b5-4975-aa1f-52e582e75e06/volumes"
Oct 30 17:54:10.408300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:10.408270    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:11.717564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:11.717530    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:12.186668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:12.186596    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:13.581989 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:13.581957    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:14.168551 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.168521    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:14.278025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.277987    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:14.308140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.308103    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"]
Oct 30 17:54:14.319823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.319796    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-043253-bpzmq"]
Oct 30 17:54:14.391040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.391008    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6922c7f-5c7a-410c-b891-e7cd46a0df17" path="/var/lib/kubelet/pods/c6922c7f-5c7a-410c-b891-e7cd46a0df17/volumes"
Oct 30 17:54:14.410245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.410215    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:14.831971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:14.831937    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:15.244768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:15.244736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:15.246772 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:15.246738    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:54:16.098183 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:16.098145    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:16.207149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:16.207112    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:17.362386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:17.362351    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:17.803093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:17.803002    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:18.746242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:18.746210    5534 scope.go:122] "RemoveContainer" containerID="b05a7dcab402648c015fff2369d22605291dc74e60e1c894e09a3eeae603a717"
Oct 30 17:54:19.279131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.279092    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"]
Oct 30 17:54:19.306764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.306724    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:19.492628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.492574    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"]
Oct 30 17:54:19.493821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.493789    5534 cpu_manager.go:414] "containerMap: removing container" podUID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" containerName="main"
Oct 30 17:54:19.493821 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.493816    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" containerName="main"
Oct 30 17:54:19.494038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.493876    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerName="payload-pre-processing"
Oct 30 17:54:19.494038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.493885    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerName="payload-pre-processing"
Oct 30 17:54:19.494152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.494068    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac24c3e-c7d1-495a-8b4d-8a87d28087a8" containerName="payload-pre-processing"
Oct 30 17:54:19.494152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.494091    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="473abcc0-5027-4e5a-8c7e-d8f46645f1ad" containerName="main"
Oct 30 17:54:19.500365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.500344    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.515069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.515041    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:54:19.515531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.515502    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:54:19.516479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.516460    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"]
Oct 30 17:54:19.595021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.594990    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.595262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.595238    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zg8k\" (UniqueName: \"kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.595339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.595315    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.635037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.635012    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:19.697731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.697692    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zg8k\" (UniqueName: \"kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.697954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.697779    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.698041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.697953    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.699340 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.699299    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.701594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.701569    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.722633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.722593    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zg8k\" (UniqueName: \"kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k\") pod \"maas-api-revoke-keys-e2e-inf-a-4a271c-ch225\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:19.815685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:19.815626    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:20.125544 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:54:20.125476    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fc9f4a_37b7_4480_9244_a2d011318dab.slice/crio-02becef587b765c0128d3e0dd1f1c014538a2f52551e1e56e088df0e73a2a61f WatchSource:0}: Error finding container 02becef587b765c0128d3e0dd1f1c014538a2f52551e1e56e088df0e73a2a61f: Status 404 returned error can't find the container with id 02becef587b765c0128d3e0dd1f1c014538a2f52551e1e56e088df0e73a2a61f
Oct 30 17:54:20.125800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:20.125777    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"]
Oct 30 17:54:21.074606 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.074567    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225" event={"ID":"28fc9f4a-37b7-4480-9244-a2d011318dab","Type":"ContainerStarted","Data":"a3504f9ede1f042f8e21cbc799c55f836f6a79c3480a6dda11b73cb755803adb"}
Oct 30 17:54:21.075189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.074663    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225" event={"ID":"28fc9f4a-37b7-4480-9244-a2d011318dab","Type":"ContainerStarted","Data":"02becef587b765c0128d3e0dd1f1c014538a2f52551e1e56e088df0e73a2a61f"}
Oct 30 17:54:21.116026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.115975    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225" podStartSLOduration=2.115959433 podStartE2EDuration="2.115959433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:21.114298196 +0000 UTC m=+2466.876342012" watchObservedRunningTime="2026-10-30 17:54:21.115959433 +0000 UTC m=+2466.878003243"
Oct 30 17:54:21.436867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.436831    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:21.551968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.551941    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:21.808731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.808575    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"]
Oct 30 17:54:21.808964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.808834    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy" containerID="cri-o://72f17919c22373d3bb79038cbde16082677b4e1b97f524ccdf040159a35c5e80" gracePeriod=30
Oct 30 17:54:21.838814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.838781    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"]
Oct 30 17:54:21.839038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.839015    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy" containerID="cri-o://331e6afe6ac25707e857e8a320dcd777371d616cf922e7c54ede1fd53403196d" gracePeriod=30
Oct 30 17:54:21.863689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.863625    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"]
Oct 30 17:54:21.863955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.863926    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy" containerID="cri-o://b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8" gracePeriod=30
Oct 30 17:54:21.871550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.871526    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"]
Oct 30 17:54:21.872028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.871771    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy" containerID="cri-o://128017967c2369d7f5763f20a10afcce20e34dfdb96a89d2ee0d86e9f74189b4" gracePeriod=30
Oct 30 17:54:21.877462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:21.877438    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:22.026378 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:22.026337    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:22.154102 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:22.154066    5534 generic.go:356] "Generic (PLEG): container finished" podID="28fc9f4a-37b7-4480-9244-a2d011318dab" containerID="a3504f9ede1f042f8e21cbc799c55f836f6a79c3480a6dda11b73cb755803adb" exitCode=0
Oct 30 17:54:22.154793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:22.154764    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225" event={"ID":"28fc9f4a-37b7-4480-9244-a2d011318dab","Type":"ContainerDied","Data":"a3504f9ede1f042f8e21cbc799c55f836f6a79c3480a6dda11b73cb755803adb"}
Oct 30 17:54:23.343671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.343621    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:23.366760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.366731    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca\") pod \"28fc9f4a-37b7-4480-9244-a2d011318dab\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") "
Oct 30 17:54:23.366901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.366847    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k\" (UniqueName: \"kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k\") pod \"28fc9f4a-37b7-4480-9244-a2d011318dab\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") "
Oct 30 17:54:23.367028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.367014    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca\") pod \"28fc9f4a-37b7-4480-9244-a2d011318dab\" (UID: \"28fc9f4a-37b7-4480-9244-a2d011318dab\") "
Oct 30 17:54:23.367077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.367022    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca" pod "28fc9f4a-37b7-4480-9244-a2d011318dab" (UID: "28fc9f4a-37b7-4480-9244-a2d011318dab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:23.367373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.367353    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:23.369161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.369134    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca" pod "28fc9f4a-37b7-4480-9244-a2d011318dab" (UID: "28fc9f4a-37b7-4480-9244-a2d011318dab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:54:23.369254 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.369213    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k" pod "28fc9f4a-37b7-4480-9244-a2d011318dab" (UID: "28fc9f4a-37b7-4480-9244-a2d011318dab"). InnerVolumeSpecName "kube-api-access-2zg8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:23.468855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.468778    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zg8k\" (UniqueName: \"kubernetes.io/projected/28fc9f4a-37b7-4480-9244-a2d011318dab-kube-api-access-2zg8k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:23.468855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.468813    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fc9f4a-37b7-4480-9244-a2d011318dab-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:23.621039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:23.621008    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:24.177105 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:24.177071    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225" event={"ID":"28fc9f4a-37b7-4480-9244-a2d011318dab","Type":"ContainerDied","Data":"02becef587b765c0128d3e0dd1f1c014538a2f52551e1e56e088df0e73a2a61f"}
Oct 30 17:54:24.177436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:24.177118    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02becef587b765c0128d3e0dd1f1c014538a2f52551e1e56e088df0e73a2a61f"
Oct 30 17:54:24.177436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:24.177174    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"
Oct 30 17:54:25.738451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:25.738418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:27.206218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.206179    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:54:27.218575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.218537    5534 generic.go:356] "Generic (PLEG): container finished" podID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerID="b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8" exitCode=0
Oct 30 17:54:27.218766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.218726    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" event={"ID":"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f","Type":"ContainerDied","Data":"b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8"}
Oct 30 17:54:27.218849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.218785    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh" event={"ID":"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f","Type":"ContainerDied","Data":"c1c0ae4ff3760f893712f2dffb2f1cf0e360c8d07488dc5ac81770563bcdd560"}
Oct 30 17:54:27.218849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.218816    5534 scope.go:122] "RemoveContainer" containerID="b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8"
Oct 30 17:54:27.219022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.218994    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"
Oct 30 17:54:27.228320 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.228285    5534 generic.go:356] "Generic (PLEG): container finished" podID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerID="72f17919c22373d3bb79038cbde16082677b4e1b97f524ccdf040159a35c5e80" exitCode=0
Oct 30 17:54:27.228530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.228503    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" event={"ID":"a694412f-e18c-4fae-99a6-e525d1cc837d","Type":"ContainerDied","Data":"72f17919c22373d3bb79038cbde16082677b4e1b97f524ccdf040159a35c5e80"}
Oct 30 17:54:27.231755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.231709    5534 generic.go:356] "Generic (PLEG): container finished" podID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerID="331e6afe6ac25707e857e8a320dcd777371d616cf922e7c54ede1fd53403196d" exitCode=0
Oct 30 17:54:27.231951 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.231924    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" event={"ID":"7b22e37f-37d9-4687-a94c-b63c842d917a","Type":"ContainerDied","Data":"331e6afe6ac25707e857e8a320dcd777371d616cf922e7c54ede1fd53403196d"}
Oct 30 17:54:27.234384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.233287    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:54:27.235822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.235794    5534 generic.go:356] "Generic (PLEG): container finished" podID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerID="128017967c2369d7f5763f20a10afcce20e34dfdb96a89d2ee0d86e9f74189b4" exitCode=0
Oct 30 17:54:27.235932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.235874    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" event={"ID":"71d71c27-0e87-45dd-8aa1-1d29ee044d5b","Type":"ContainerDied","Data":"128017967c2369d7f5763f20a10afcce20e34dfdb96a89d2ee0d86e9f74189b4"}
Oct 30 17:54:27.289691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.289609    5534 scope.go:122] "RemoveContainer" containerID="4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"
Oct 30 17:54:27.337924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.336826    5534 scope.go:122] "RemoveContainer" containerID="b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8"
Oct 30 17:54:27.344435 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:27.344045    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8\": container with ID starting with b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8 not found: ID does not exist" containerID="b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8"
Oct 30 17:54:27.344435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.344089    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8"} err="failed to get container status \"b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8\": rpc error: code = NotFound desc = could not find container \"b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8\": container with ID starting with b265a5a1244398d0ddbc89c28bf1a8c6731d8ed4a9a06cfdccae020edc83d7b8 not found: ID does not exist"
Oct 30 17:54:27.344435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.344115    5534 scope.go:122] "RemoveContainer" containerID="4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"
Oct 30 17:54:27.345359 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:27.344989    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da\": container with ID starting with 4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da not found: ID does not exist" containerID="4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"
Oct 30 17:54:27.345359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.345023    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da"} err="failed to get container status \"4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da\": rpc error: code = NotFound desc = could not find container \"4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da\": container with ID starting with 4239f9af47d624eb6f239579367e60748400ec9ff38796ebcafe2db4032ce2da not found: ID does not exist"
Oct 30 17:54:27.346613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346587    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.346782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346665    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.346782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346699    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.346782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346739    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.346973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346850    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.346973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346887    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.346965    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.347098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347045    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.347098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347078    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.347257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347117    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347159    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347257 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347216    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347256    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.347427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347305    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347347    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.347427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347388    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347416    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347496    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert\") pod \"7b22e37f-37d9-4687-a94c-b63c842d917a\" (UID: \"7b22e37f-37d9-4687-a94c-b63c842d917a\") "
Oct 30 17:54:27.347735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347563    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data\") pod \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\" (UID: \"1d1e600c-ac6e-4793-bece-3ae2e1c4e45f\") "
Oct 30 17:54:27.347735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.347551    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.352784    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.356079    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.356387    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.356437    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.357107    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.357328    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.360324    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:27.362025 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.361516    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "kube-api-access-9fqwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.363004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.362974    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.363144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.363009    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.363144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.363023    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.363144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.363036    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.363456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.363420    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:54:27.363568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.363547    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.363660 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.363611    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.366695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.366597    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.366915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.366826    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.368117 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.368091    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk" pod "7b22e37f-37d9-4687-a94c-b63c842d917a" (UID: "7b22e37f-37d9-4687-a94c-b63c842d917a"). InnerVolumeSpecName "kube-api-access-p76dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.368441 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.368419    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.369871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.368833    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:54:27.371423 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.371402    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:54:27.374396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.374376    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo" pod "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" (UID: "1d1e600c-ac6e-4793-bece-3ae2e1c4e45f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:54:27.464095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464040    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.464309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464228    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.464309 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464282    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.464447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464337    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.464447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464374    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.464447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464410    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.464610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464445    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.464610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464518    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.464768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464615    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.464768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464617    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:27.464768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464695    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.464768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464700    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.464768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464733    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464771    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464812    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464866    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464904    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464950    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo\") pod \"a694412f-e18c-4fae-99a6-e525d1cc837d\" (UID: \"a694412f-e18c-4fae-99a6-e525d1cc837d\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.464997    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.465042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.465014    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.465393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.465047    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token\") pod \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\" (UID: \"71d71c27-0e87-45dd-8aa1-1d29ee044d5b\") "
Oct 30 17:54:27.465393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.465169    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.465393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.465203    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.465560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.465392    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.465613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.465561    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.467432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.467404    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.467944    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468117    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468136    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468151    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468165    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468178    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468191    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a694412f-e18c-4fae-99a6-e525d1cc837d-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468204    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468217    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468220    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:27.468251 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468230    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468261    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqwz\" (UniqueName: \"kubernetes.io/projected/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-kube-api-access-9fqwz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468278    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468293    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468309    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468325    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468341    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468356    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22e37f-37d9-4687-a94c-b63c842d917a-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468370    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468385    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22e37f-37d9-4687-a94c-b63c842d917a-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468400    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76dk\" (UniqueName: \"kubernetes.io/projected/7b22e37f-37d9-4687-a94c-b63c842d917a-kube-api-access-p76dk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468415    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22e37f-37d9-4687-a94c-b63c842d917a-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468434    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468448    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468477    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.468805    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:54:27.469367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.469182    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:54:27.471456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.469682    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "kube-api-access-zxsj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.471456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.470230    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.476215 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.476175    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.476455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.476432    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token" pod "71d71c27-0e87-45dd-8aa1-1d29ee044d5b" (UID: "71d71c27-0e87-45dd-8aa1-1d29ee044d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.478451 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.478235    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:27.478712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.478689    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng" pod "a694412f-e18c-4fae-99a6-e525d1cc837d" (UID: "a694412f-e18c-4fae-99a6-e525d1cc837d"). InnerVolumeSpecName "kube-api-access-g4wng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:27.569287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569253    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"]
Oct 30 17:54:27.569801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569779    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569805    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxsj7\" (UniqueName: \"kubernetes.io/projected/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-kube-api-access-zxsj7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569817    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569827    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569835    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569843    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4wng\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-kube-api-access-g4wng\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569850    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d71c27-0e87-45dd-8aa1-1d29ee044d5b-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.569858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569859    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.570094 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.569867    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a694412f-e18c-4fae-99a6-e525d1cc837d-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:27.577616 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.577588    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bczqh"]
Oct 30 17:54:27.652490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:27.652458    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:28.108460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.108436    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:28.245398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.245369    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"
Oct 30 17:54:28.245398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.245391    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6" event={"ID":"a694412f-e18c-4fae-99a6-e525d1cc837d","Type":"ContainerDied","Data":"2e5bfe3405970cee2933426642f923eb2f38c63f953546a230676304a79eb602"}
Oct 30 17:54:28.245970 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.245470    5534 scope.go:122] "RemoveContainer" containerID="72f17919c22373d3bb79038cbde16082677b4e1b97f524ccdf040159a35c5e80"
Oct 30 17:54:28.247258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.247242    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"
Oct 30 17:54:28.247355 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.247328    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5" event={"ID":"7b22e37f-37d9-4687-a94c-b63c842d917a","Type":"ContainerDied","Data":"c5a4c23f2c12be7eccbf0aa851a8cd86b7f1bf6809525885e65d2ef526a0af91"}
Oct 30 17:54:28.248962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.248936    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"
Oct 30 17:54:28.249091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.248939    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m" event={"ID":"71d71c27-0e87-45dd-8aa1-1d29ee044d5b","Type":"ContainerDied","Data":"05abc52fc884a082814defdc48a7b1d901f87da84dcba51c5929b7aabc58f9bf"}
Oct 30 17:54:28.259295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.259277    5534 scope.go:122] "RemoveContainer" containerID="331e6afe6ac25707e857e8a320dcd777371d616cf922e7c54ede1fd53403196d"
Oct 30 17:54:28.270343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.270324    5534 scope.go:122] "RemoveContainer" containerID="128017967c2369d7f5763f20a10afcce20e34dfdb96a89d2ee0d86e9f74189b4"
Oct 30 17:54:28.319469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.319439    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"]
Oct 30 17:54:28.332881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.332848    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-dpkj6"]
Oct 30 17:54:28.360617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.360530    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"]
Oct 30 17:54:28.371557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.371524    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" path="/var/lib/kubelet/pods/1d1e600c-ac6e-4793-bece-3ae2e1c4e45f/volumes"
Oct 30 17:54:28.372206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.372187    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" path="/var/lib/kubelet/pods/a694412f-e18c-4fae-99a6-e525d1cc837d/volumes"
Oct 30 17:54:28.372615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.372598    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-g2fk5"]
Oct 30 17:54:28.397165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.397129    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"]
Oct 30 17:54:28.406267 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.406236    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5bm5m"]
Oct 30 17:54:28.582044 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:28.582006    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:29.016625 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.016593    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"]
Oct 30 17:54:29.017666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017616    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy"
Oct 30 17:54:29.017666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017660    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy"
Oct 30 17:54:29.017777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017691    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy"
Oct 30 17:54:29.017777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017699    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy"
Oct 30 17:54:29.017777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017724    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy"
Oct 30 17:54:29.017777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017733    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy"
Oct 30 17:54:29.017777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017759    5534 cpu_manager.go:414] "containerMap: removing container" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy"
Oct 30 17:54:29.017777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017766    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy"
Oct 30 17:54:29.018003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017806    5534 cpu_manager.go:414] "containerMap: removing container" podUID="28fc9f4a-37b7-4480-9244-a2d011318dab" containerName="revoke-keys"
Oct 30 17:54:29.018003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017813    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fc9f4a-37b7-4480-9244-a2d011318dab" containerName="revoke-keys"
Oct 30 17:54:29.018003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017825    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy"
Oct 30 17:54:29.018003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.017833    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy"
Oct 30 17:54:29.018140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.018026    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" containerName="istio-proxy"
Oct 30 17:54:29.018140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.018045    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy"
Oct 30 17:54:29.018140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.018058    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e600c-ac6e-4793-bece-3ae2e1c4e45f" containerName="istio-proxy"
Oct 30 17:54:29.018140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.018069    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" containerName="istio-proxy"
Oct 30 17:54:29.018140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.018083    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fc9f4a-37b7-4480-9244-a2d011318dab" containerName="revoke-keys"
Oct 30 17:54:29.018140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.018093    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a694412f-e18c-4fae-99a6-e525d1cc837d" containerName="istio-proxy"
Oct 30 17:54:29.024549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.024510    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.043596 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.043577    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-d207f8-openshift-default-dockercfg-d2z4p\""
Oct 30 17:54:29.052083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.052057    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"]
Oct 30 17:54:29.092240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092203    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092399 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092370    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092502    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092626 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092608    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092708    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdd2\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092748    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092881    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092918    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.092972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.092937    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.105834 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.105807    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:29.212669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212227    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.212669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212432    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.212669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212524    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.212669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212625    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.213067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212717    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdd2\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.213067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212764    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.213067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212885    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.213067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212934    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.213067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.212994    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.221665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.213672    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.221665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.213818    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.221665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.214046    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.221665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.214362    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.257896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.230949    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.257896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.231995    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.270033 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.268366    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdd2\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.276713 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.276687    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.278749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.278615    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.343470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.343437    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:29.411723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.411510    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:29.685467 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.685440    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"]
Oct 30 17:54:29.685622 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:54:29.685595    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e416bc_4744_4354_b782_9ae4e6c64217.slice/crio-e9d22f12cb4a73c163fddc4f268c709c7ea379a0843b768a1f5e226393e6fabd WatchSource:0}: Error finding container e9d22f12cb4a73c163fddc4f268c709c7ea379a0843b768a1f5e226393e6fabd: Status 404 returned error can't find the container with id e9d22f12cb4a73c163fddc4f268c709c7ea379a0843b768a1f5e226393e6fabd
Oct 30 17:54:29.688980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.688949    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:54:29.689092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.689008    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:54:29.689092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.689039    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:54:29.727173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:29.727137    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:30.013862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.013781    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"]
Oct 30 17:54:30.060539 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.060489    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"]
Oct 30 17:54:30.168620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.168580    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:30.293862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.293749    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:30.329273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.329237    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" event={"ID":"88e416bc-4744-4354-b782-9ae4e6c64217","Type":"ContainerStarted","Data":"2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"}
Oct 30 17:54:30.329686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.329329    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" event={"ID":"88e416bc-4744-4354-b782-9ae4e6c64217","Type":"ContainerStarted","Data":"e9d22f12cb4a73c163fddc4f268c709c7ea379a0843b768a1f5e226393e6fabd"}
Oct 30 17:54:30.344568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.344441    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:30.351899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.351876    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:30.379881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.374517    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d71c27-0e87-45dd-8aa1-1d29ee044d5b" path="/var/lib/kubelet/pods/71d71c27-0e87-45dd-8aa1-1d29ee044d5b/volumes"
Oct 30 17:54:30.379881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.375284    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b22e37f-37d9-4687-a94c-b63c842d917a" path="/var/lib/kubelet/pods/7b22e37f-37d9-4687-a94c-b63c842d917a/volumes"
Oct 30 17:54:30.384716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:30.384632    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podStartSLOduration=2.384618319 podStartE2EDuration="2.384618319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:30.382797176 +0000 UTC m=+2476.144841008" watchObservedRunningTime="2026-10-30 17:54:30.384618319 +0000 UTC m=+2476.146662132"
Oct 30 17:54:31.338349 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:31.338320    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:31.339204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:31.339186    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:54:31.738024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:31.737989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:32.837040 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:32.837000    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"]
Oct 30 17:54:32.866324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:32.866292    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:33.007421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:33.007389    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:33.047997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:33.047966    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"]
Oct 30 17:54:33.051289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:33.051265    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9ad10-wdwv2"]
Oct 30 17:54:33.207673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:33.207241    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:33.733462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:33.733431    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:34.307134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.307083    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw" podUID="256d788e-3a50-4a0d-87bc-95567f350332" containerName="main" containerID="cri-o://fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75" gracePeriod=45
Oct 30 17:54:34.377490 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.377459    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ff5baf-45f4-4742-9d01-0c290eb06231" path="/var/lib/kubelet/pods/89ff5baf-45f4-4742-9d01-0c290eb06231/volumes"
Oct 30 17:54:34.606186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.606160    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652234    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache\") pod \"256d788e-3a50-4a0d-87bc-95567f350332\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") "
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652297    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm\") pod \"256d788e-3a50-4a0d-87bc-95567f350332\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") "
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652341    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs\" (UniqueName: \"kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs\") pod \"256d788e-3a50-4a0d-87bc-95567f350332\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") "
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652476    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir\") pod \"256d788e-3a50-4a0d-87bc-95567f350332\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") "
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652576    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh\" (UniqueName: \"kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh\") pod \"256d788e-3a50-4a0d-87bc-95567f350332\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") "
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652576    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache" pod "256d788e-3a50-4a0d-87bc-95567f350332" (UID: "256d788e-3a50-4a0d-87bc-95567f350332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.652818    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir" pod "256d788e-3a50-4a0d-87bc-95567f350332" (UID: "256d788e-3a50-4a0d-87bc-95567f350332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:34.654668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.653761    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home\") pod \"256d788e-3a50-4a0d-87bc-95567f350332\" (UID: \"256d788e-3a50-4a0d-87bc-95567f350332\") "
Oct 30 17:54:34.655238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.654894    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:34.655238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.654915    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:34.655238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.655186    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home" pod "256d788e-3a50-4a0d-87bc-95567f350332" (UID: "256d788e-3a50-4a0d-87bc-95567f350332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:34.657096 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.655831    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs" pod "256d788e-3a50-4a0d-87bc-95567f350332" (UID: "256d788e-3a50-4a0d-87bc-95567f350332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:54:34.657870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.657829    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh" pod "256d788e-3a50-4a0d-87bc-95567f350332" (UID: "256d788e-3a50-4a0d-87bc-95567f350332"). InnerVolumeSpecName "kube-api-access-xgjbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:34.682109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.669827    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm" pod "256d788e-3a50-4a0d-87bc-95567f350332" (UID: "256d788e-3a50-4a0d-87bc-95567f350332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:54:34.727161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.727077    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:34.756364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.756337    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d788e-3a50-4a0d-87bc-95567f350332-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:34.756364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.756364    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjbh\" (UniqueName: \"kubernetes.io/projected/256d788e-3a50-4a0d-87bc-95567f350332-kube-api-access-xgjbh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:34.756573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.756373    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:34.756573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.756383    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d788e-3a50-4a0d-87bc-95567f350332-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:34.842561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.842532    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:34.894801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.894773    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"]
Oct 30 17:54:34.895493 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.895476    5534 cpu_manager.go:414] "containerMap: removing container" podUID="256d788e-3a50-4a0d-87bc-95567f350332" containerName="main"
Oct 30 17:54:34.895580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.895494    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256d788e-3a50-4a0d-87bc-95567f350332" containerName="main"
Oct 30 17:54:34.895662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.895635    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="256d788e-3a50-4a0d-87bc-95567f350332" containerName="main"
Oct 30 17:54:34.899545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.899524    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:34.944150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.941739    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"]
Oct 30 17:54:34.959687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.945516    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-d207f8\""
Oct 30 17:54:34.959687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.945672    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-d207f8-dockercfg-p55p9\""
Oct 30 17:54:34.971453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.965718    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjr6\" (UniqueName: \"kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6\") pod \"payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:34.971453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:34.966083    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:35.110794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.110705    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerName="payload-processing" containerID="cri-o://b79a52e2e38ff51d766d97dcd324ed7c8c4ebeb448a0eb21057480a3662a6988" gracePeriod=25
Oct 30 17:54:35.118698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.111763    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerName="payload-pre-processing" containerID="cri-o://c4bdedaec9badfc70007f908bcbe27140dc11b274c8a588769e36e233cdcbff0" gracePeriod=25
Oct 30 17:54:35.118698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.112184    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:35.118698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.115105    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjr6\" (UniqueName: \"kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6\") pod \"payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:35.118698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.113170    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:35.138751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.138723    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjr6\" (UniqueName: \"kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6\") pod \"payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:35.214963 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.214924    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:35.379388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.379287    5534 generic.go:356] "Generic (PLEG): container finished" podID="256d788e-3a50-4a0d-87bc-95567f350332" containerID="fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75" exitCode=0
Oct 30 17:54:35.379388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.379333    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"
Oct 30 17:54:35.379388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.379361    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw" event={"ID":"256d788e-3a50-4a0d-87bc-95567f350332","Type":"ContainerDied","Data":"fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75"}
Oct 30 17:54:35.379939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.379406    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw" event={"ID":"256d788e-3a50-4a0d-87bc-95567f350332","Type":"ContainerDied","Data":"3fb24a4427f7e3704b4fd14d588b37a8effbbe50cb3ccf72ce76777851f4afe5"}
Oct 30 17:54:35.379939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.379425    5534 scope.go:122] "RemoveContainer" containerID="fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75"
Oct 30 17:54:35.391747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.391724    5534 scope.go:122] "RemoveContainer" containerID="fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75"
Oct 30 17:54:35.392086 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:35.392062    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75\": container with ID starting with fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75 not found: ID does not exist" containerID="fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75"
Oct 30 17:54:35.392192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.392092    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75"} err="failed to get container status \"fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75\": rpc error: code = NotFound desc = could not find container \"fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75\": container with ID starting with fb8bac76e0998a52320eae5dd4c1cbbacaef167d375c4adf939e15f363e81a75 not found: ID does not exist"
Oct 30 17:54:35.431662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.431605    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"]
Oct 30 17:54:35.436045 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.436022    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c596f1/test-model-c596f1-kserve-5c759849bc-hmpsw"]
Oct 30 17:54:35.532887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.532849    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"]
Oct 30 17:54:35.533316 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:54:35.533295    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f35eae8_b2d2_4c94_98e4_de92f0efa592.slice/crio-4123b4ca71b22119f198bc8f8a1e52bcade44360383237c97f118a028950f6d0 WatchSource:0}: Error finding container 4123b4ca71b22119f198bc8f8a1e52bcade44360383237c97f118a028950f6d0: Status 404 returned error can't find the container with id 4123b4ca71b22119f198bc8f8a1e52bcade44360383237c97f118a028950f6d0
Oct 30 17:54:35.953744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:35.953692    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:36.372324 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:36.372288    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256d788e-3a50-4a0d-87bc-95567f350332" path="/var/lib/kubelet/pods/256d788e-3a50-4a0d-87bc-95567f350332/volumes"
Oct 30 17:54:36.390997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:36.390960    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" event={"ID":"7f35eae8-b2d2-4c94-98e4-de92f0efa592","Type":"ContainerStarted","Data":"b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d"}
Oct 30 17:54:36.391461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:36.391027    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" event={"ID":"7f35eae8-b2d2-4c94-98e4-de92f0efa592","Type":"ContainerStarted","Data":"4123b4ca71b22119f198bc8f8a1e52bcade44360383237c97f118a028950f6d0"}
Oct 30 17:54:36.391461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:36.391056    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:36.434962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:36.434915    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" podStartSLOduration=2.434902204 podStartE2EDuration="2.434902204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:36.430783518 +0000 UTC m=+2482.192827332" watchObservedRunningTime="2026-10-30 17:54:36.434902204 +0000 UTC m=+2482.196946016"
Oct 30 17:54:37.837571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:37.837495    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc" podUID="759e7c9d-045a-4b10-8ffb-c5773e6e3334" containerName="payload-processing" containerID="cri-o://95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808" gracePeriod=25
Oct 30 17:54:38.054687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.054502    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:38.144472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.144449    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:54:38.275185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.275143    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t\" (UniqueName: \"kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t\") pod \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") "
Oct 30 17:54:38.275400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.275195    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume\") pod \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\" (UID: \"759e7c9d-045a-4b10-8ffb-c5773e6e3334\") "
Oct 30 17:54:38.275734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.275694    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume" pod "759e7c9d-045a-4b10-8ffb-c5773e6e3334" (UID: "759e7c9d-045a-4b10-8ffb-c5773e6e3334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:38.277735 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.277706    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t" pod "759e7c9d-045a-4b10-8ffb-c5773e6e3334" (UID: "759e7c9d-045a-4b10-8ffb-c5773e6e3334"). InnerVolumeSpecName "kube-api-access-5zd7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:38.376817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.376740    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zd7t\" (UniqueName: \"kubernetes.io/projected/759e7c9d-045a-4b10-8ffb-c5773e6e3334-kube-api-access-5zd7t\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:38.376817 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.376776    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759e7c9d-045a-4b10-8ffb-c5773e6e3334-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:38.410529 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.410491    5534 generic.go:356] "Generic (PLEG): container finished" podID="759e7c9d-045a-4b10-8ffb-c5773e6e3334" containerID="95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808" exitCode=0
Oct 30 17:54:38.410752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.410541    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"
Oct 30 17:54:38.410752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.410568    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc" event={"ID":"759e7c9d-045a-4b10-8ffb-c5773e6e3334","Type":"ContainerDied","Data":"95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808"}
Oct 30 17:54:38.410752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.410631    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc" event={"ID":"759e7c9d-045a-4b10-8ffb-c5773e6e3334","Type":"ContainerDied","Data":"324826724f4deda751bcca68010c78541ae1cb01c3e7a36492615d5c137d71d4"}
Oct 30 17:54:38.410752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.410678    5534 scope.go:122] "RemoveContainer" containerID="95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808"
Oct 30 17:54:38.421889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.421867    5534 scope.go:122] "RemoveContainer" containerID="95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808"
Oct 30 17:54:38.422158 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:38.422134    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808\": container with ID starting with 95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808 not found: ID does not exist" containerID="95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808"
Oct 30 17:54:38.422221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.422171    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808"} err="failed to get container status \"95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808\": rpc error: code = NotFound desc = could not find container \"95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808\": container with ID starting with 95b15c6481eab8ec98cee0e792a55d46d1c12c963934e5fcf75dfbcedb02d808 not found: ID does not exist"
Oct 30 17:54:38.484836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.484797    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"]
Oct 30 17:54:38.494240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:38.494211    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bcdac0-7d8c8c57c6-zqdxc"]
Oct 30 17:54:39.890959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:39.890927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:40.373089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:40.373058    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759e7c9d-045a-4b10-8ffb-c5773e6e3334" path="/var/lib/kubelet/pods/759e7c9d-045a-4b10-8ffb-c5773e6e3334/volumes"
Oct 30 17:54:41.725043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:41.725004    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:42.344088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:42.344049    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:54:42.344261 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:42.344067    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:54:43.725401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:43.725371    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:45.245356 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:54:45.245315    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:54:45.652077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:45.652029    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:47.405012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:47.404984    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 17:54:47.544093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:47.544057    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:49.216923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:49.216890    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:49.962111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:49.962071    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:51.156669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:51.156209    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:52.219351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:52.219320    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:52.344134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:52.344098    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:54:52.344351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:52.344299    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:54:53.286666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:53.279406    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:55.047265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.047234    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:55.408631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.405125    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:55.694871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.693943    5534 generic.go:356] "Generic (PLEG): container finished" podID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerID="b79a52e2e38ff51d766d97dcd324ed7c8c4ebeb448a0eb21057480a3662a6988" exitCode=0
Oct 30 17:54:55.694871 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.694176    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" event={"ID":"1045076d-f452-4b16-95d8-d5e2e388b9e8","Type":"ContainerDied","Data":"b79a52e2e38ff51d766d97dcd324ed7c8c4ebeb448a0eb21057480a3662a6988"}
Oct 30 17:54:55.701698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.696365    5534 generic.go:356] "Generic (PLEG): container finished" podID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerID="c4bdedaec9badfc70007f908bcbe27140dc11b274c8a588769e36e233cdcbff0" exitCode=0
Oct 30 17:54:55.701698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.696553    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" event={"ID":"2668aea7-9fbc-43c0-beec-eb6ce300ce35","Type":"ContainerDied","Data":"c4bdedaec9badfc70007f908bcbe27140dc11b274c8a588769e36e233cdcbff0"}
Oct 30 17:54:55.701698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.696599    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f" event={"ID":"2668aea7-9fbc-43c0-beec-eb6ce300ce35","Type":"ContainerDied","Data":"a0b1f468818fa9d70a3b025125b03540e1605f5014fc82672486fc120602e9b5"}
Oct 30 17:54:55.701698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.696625    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b1f468818fa9d70a3b025125b03540e1605f5014fc82672486fc120602e9b5"
Oct 30 17:54:55.701698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.698537    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:54:55.712785 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.712760    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:54:55.841995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.841959    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr\" (UniqueName: \"kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr\") pod \"1045076d-f452-4b16-95d8-d5e2e388b9e8\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") "
Oct 30 17:54:55.842204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.842005    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume\") pod \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") "
Oct 30 17:54:55.842204 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.842070    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume\") pod \"1045076d-f452-4b16-95d8-d5e2e388b9e8\" (UID: \"1045076d-f452-4b16-95d8-d5e2e388b9e8\") "
Oct 30 17:54:55.842353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.842327    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj\" (UniqueName: \"kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj\") pod \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\" (UID: \"2668aea7-9fbc-43c0-beec-eb6ce300ce35\") "
Oct 30 17:54:55.842430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.842399    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume" pod "2668aea7-9fbc-43c0-beec-eb6ce300ce35" (UID: "2668aea7-9fbc-43c0-beec-eb6ce300ce35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:55.842516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.842496    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume" pod "1045076d-f452-4b16-95d8-d5e2e388b9e8" (UID: "1045076d-f452-4b16-95d8-d5e2e388b9e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:55.843078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.843040    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2668aea7-9fbc-43c0-beec-eb6ce300ce35-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:55.843078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.843063    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1045076d-f452-4b16-95d8-d5e2e388b9e8-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:55.844367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.844345    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr" pod "1045076d-f452-4b16-95d8-d5e2e388b9e8" (UID: "1045076d-f452-4b16-95d8-d5e2e388b9e8"). InnerVolumeSpecName "kube-api-access-ltvqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:55.844484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.844422    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj" pod "2668aea7-9fbc-43c0-beec-eb6ce300ce35" (UID: "2668aea7-9fbc-43c0-beec-eb6ce300ce35"). InnerVolumeSpecName "kube-api-access-lbscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:55.944166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.944085    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbscj\" (UniqueName: \"kubernetes.io/projected/2668aea7-9fbc-43c0-beec-eb6ce300ce35-kube-api-access-lbscj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:55.944166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:55.944112    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvqr\" (UniqueName: \"kubernetes.io/projected/1045076d-f452-4b16-95d8-d5e2e388b9e8-kube-api-access-ltvqr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:54:56.712759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.712619    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"
Oct 30 17:54:56.712759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.712636    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps" event={"ID":"1045076d-f452-4b16-95d8-d5e2e388b9e8","Type":"ContainerDied","Data":"aaa42f687ba813ea95ab43422f13fb835bc71e5365db01e8ba9bb89bdb011d00"}
Oct 30 17:54:56.712759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.712744    5534 scope.go:122] "RemoveContainer" containerID="b79a52e2e38ff51d766d97dcd324ed7c8c4ebeb448a0eb21057480a3662a6988"
Oct 30 17:54:56.713388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.712844    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"
Oct 30 17:54:56.792259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.792208    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"]
Oct 30 17:54:56.796886 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.796860    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a271c-5cf5cf99-47z4f"]
Oct 30 17:54:56.804753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.804730    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:56.824822 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.824795    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"]
Oct 30 17:54:56.828561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:56.828538    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a271c-5b5b5c749b-tjdps"]
Oct 30 17:54:58.383489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.383446    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" path="/var/lib/kubelet/pods/1045076d-f452-4b16-95d8-d5e2e388b9e8/volumes"
Oct 30 17:54:58.392705 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.392624    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" path="/var/lib/kubelet/pods/2668aea7-9fbc-43c0-beec-eb6ce300ce35/volumes"
Oct 30 17:54:58.731211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.731126    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:58.931100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.931064    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"]
Oct 30 17:54:58.932022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.931994    5534 cpu_manager.go:414] "containerMap: removing container" podUID="759e7c9d-045a-4b10-8ffb-c5773e6e3334" containerName="payload-processing"
Oct 30 17:54:58.932022 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932021    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759e7c9d-045a-4b10-8ffb-c5773e6e3334" containerName="payload-processing"
Oct 30 17:54:58.932242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932047    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerName="payload-pre-processing"
Oct 30 17:54:58.932242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932055    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerName="payload-pre-processing"
Oct 30 17:54:58.932242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932088    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerName="payload-processing"
Oct 30 17:54:58.932242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932097    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerName="payload-processing"
Oct 30 17:54:58.932477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932260    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="759e7c9d-045a-4b10-8ffb-c5773e6e3334" containerName="payload-processing"
Oct 30 17:54:58.932477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932276    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2668aea7-9fbc-43c0-beec-eb6ce300ce35" containerName="payload-pre-processing"
Oct 30 17:54:58.932477 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.932291    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1045076d-f452-4b16-95d8-d5e2e388b9e8" containerName="payload-processing"
Oct 30 17:54:58.938177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.938136    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:58.960409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.960381    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"]
Oct 30 17:54:58.965815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.965779    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"auth-test-model-a2a6cb-kserve-self-signed-certs\""
Oct 30 17:54:58.965978 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.965897    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"default-dockercfg-tsvtz\""
Oct 30 17:54:58.969923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.969883    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"openshift-service-ca.crt\""
Oct 30 17:54:58.970072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.969965    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"kube-root-ca.crt\""
Oct 30 17:54:58.979151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:58.978346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:54:59.122537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.122497    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.122775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.122588    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.122775 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.122730    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.122869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.122816    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.122973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.122959    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.123034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.123011    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxkc\" (UniqueName: \"kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.225561    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.225655    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxkc\" (UniqueName: \"kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.225761    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.225870    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.225904    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.225965    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.226443    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.227379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.226695    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.232677 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.228471    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.237758 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.235155    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.240760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.240728    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.251813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.251758    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxkc\" (UniqueName: \"kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc\") pod \"auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57\" (UID: \"4690a357-7337-4083-814c-690361db774e\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.266679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.261766    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:54:59.582780 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.582690    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"]
Oct 30 17:54:59.744842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.744807    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57" event={"ID":"4690a357-7337-4083-814c-690361db774e","Type":"ContainerStarted","Data":"08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27"}
Oct 30 17:54:59.745046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.744858    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57" event={"ID":"4690a357-7337-4083-814c-690361db774e","Type":"ContainerStarted","Data":"b54c29504125d70e90148af216b6260e6dc9b481a3ad20f2fb6caee4894c3779"}
Oct 30 17:54:59.801782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.801728    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57" podStartSLOduration=1.801713925 podStartE2EDuration="1.801713925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:59.797495848 +0000 UTC m=+2505.559539662" watchObservedRunningTime="2026-10-30 17:54:59.801713925 +0000 UTC m=+2505.563757737"
Oct 30 17:54:59.858811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.858779    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:54:59.859008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:54:59.858836    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:55:00.156707 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.156576    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"]
Oct 30 17:55:00.156950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.156841    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk" podUID="79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" containerName="maas-api" containerID="cri-o://346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238" gracePeriod=30
Oct 30 17:55:00.237426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.237390    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:00.262751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.262691    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:55:00.554186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.554141    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:55:00.653666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.650785    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs\") pod \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") "
Oct 30 17:55:00.653666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.650863    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz\" (UniqueName: \"kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz\") pod \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") "
Oct 30 17:55:00.653666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.651186    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca\") pod \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") "
Oct 30 17:55:00.653666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.651227    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls\") pod \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\" (UID: \"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d\") "
Oct 30 17:55:00.682668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.661739    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca" pod "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" (UID: "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:00.682668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.673808    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz" pod "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" (UID: "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d"). InnerVolumeSpecName "kube-api-access-bcbnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:00.762054 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.761711    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:00.762343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.762322    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbnz\" (UniqueName: \"kubernetes.io/projected/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-kube-api-access-bcbnz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:00.766380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.766301    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs" pod "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" (UID: "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:00.773866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.766745    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls" pod "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" (UID: "79b6d9c6-5bf7-446d-9261-ae9d8672aa2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:00.792446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.792409    5534 generic.go:356] "Generic (PLEG): container finished" podID="79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" containerID="346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238" exitCode=0
Oct 30 17:55:00.792723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.792471    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"
Oct 30 17:55:00.792723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.792490    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk" event={"ID":"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d","Type":"ContainerDied","Data":"346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238"}
Oct 30 17:55:00.792723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.792545    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk" event={"ID":"79b6d9c6-5bf7-446d-9261-ae9d8672aa2d","Type":"ContainerDied","Data":"0d86d00127260c44350b5a4ee00dbcdb4cb028863652af3f2c0c737e891dd002"}
Oct 30 17:55:00.792723 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.792570    5534 scope.go:122] "RemoveContainer" containerID="346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238"
Oct 30 17:55:00.807407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.807387    5534 scope.go:122] "RemoveContainer" containerID="346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238"
Oct 30 17:55:00.807784 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:00.807761    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238\": container with ID starting with 346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238 not found: ID does not exist" containerID="346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238"
Oct 30 17:55:00.807875 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.807795    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238"} err="failed to get container status \"346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238\": rpc error: code = NotFound desc = could not find container \"346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238\": container with ID starting with 346954ae31e761463f0938870386819eccf48e0516d08c0a73049c9ddfe1b238 not found: ID does not exist"
Oct 30 17:55:00.829522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.829489    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:00.844656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.844567    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"]
Oct 30 17:55:00.849219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.849186    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcdac0-668b567fdf-9wxlk"]
Oct 30 17:55:00.863954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.863915    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:00.863954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.863956    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:00.978595 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:00.978561    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:01.104283 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:01.104249    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:01.507549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:01.507428    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:02.377888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.377852    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" path="/var/lib/kubelet/pods/79b6d9c6-5bf7-446d-9261-ae9d8672aa2d/volumes"
Oct 30 17:55:02.815581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.815545    5534 generic.go:356] "Generic (PLEG): container finished" podID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerID="e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3" exitCode=0
Oct 30 17:55:02.815812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.815621    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" event={"ID":"07bfc8c2-a073-4c16-b984-282c82d720e5","Type":"ContainerDied","Data":"e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"}
Oct 30 17:55:02.816146 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.816128    5534 scope.go:122] "RemoveContainer" containerID="e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"
Oct 30 17:55:02.817894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.817866    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:02.817981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.817935    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:02.817981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:02.817968    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:03.150453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:03.150418    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:03.826667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:03.826595    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" event={"ID":"07bfc8c2-a073-4c16-b984-282c82d720e5","Type":"ContainerStarted","Data":"ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278"}
Oct 30 17:55:04.766669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:04.766095    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:55:04.766669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:04.766238    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 17:55:04.766669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:04.766283    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 17:55:04.936430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:04.936397    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:05.767666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:05.763083    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 17:55:05.767666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:05.763149    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 17:55:06.441702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:06.441569    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"]
Oct 30 17:55:06.442226 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:06.441843    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh" podUID="b2601a6f-c711-41c0-9620-f311e6ba91d5" containerName="maas-api" containerID="cri-o://1a44dad4a3a27ae0d24d16f50f31589429215f80d69197d6c1d6d4620b5f23e4" gracePeriod=30
Oct 30 17:55:06.536589 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:06.536311    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:06.800865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:06.797071    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 17:55:06.800865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:06.797150    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 17:55:07.124762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.124723    5534 generic.go:356] "Generic (PLEG): container finished" podID="b2601a6f-c711-41c0-9620-f311e6ba91d5" containerID="1a44dad4a3a27ae0d24d16f50f31589429215f80d69197d6c1d6d4620b5f23e4" exitCode=0
Oct 30 17:55:07.125038 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.125017    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh" event={"ID":"b2601a6f-c711-41c0-9620-f311e6ba91d5","Type":"ContainerDied","Data":"1a44dad4a3a27ae0d24d16f50f31589429215f80d69197d6c1d6d4620b5f23e4"}
Oct 30 17:55:07.137126 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.137098    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:55:07.248297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.248256    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:07.283667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.278196    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls\") pod \"b2601a6f-c711-41c0-9620-f311e6ba91d5\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") "
Oct 30 17:55:07.283667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.278315    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs\") pod \"b2601a6f-c711-41c0-9620-f311e6ba91d5\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") "
Oct 30 17:55:07.283667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.278386    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca\") pod \"b2601a6f-c711-41c0-9620-f311e6ba91d5\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") "
Oct 30 17:55:07.283667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.278481    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96\" (UniqueName: \"kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96\") pod \"b2601a6f-c711-41c0-9620-f311e6ba91d5\" (UID: \"b2601a6f-c711-41c0-9620-f311e6ba91d5\") "
Oct 30 17:55:07.283667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.280556    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca" pod "b2601a6f-c711-41c0-9620-f311e6ba91d5" (UID: "b2601a6f-c711-41c0-9620-f311e6ba91d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:07.302694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.302235    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs" pod "b2601a6f-c711-41c0-9620-f311e6ba91d5" (UID: "b2601a6f-c711-41c0-9620-f311e6ba91d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:07.303141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.303109    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls" pod "b2601a6f-c711-41c0-9620-f311e6ba91d5" (UID: "b2601a6f-c711-41c0-9620-f311e6ba91d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:07.304592 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.304131    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96" pod "b2601a6f-c711-41c0-9620-f311e6ba91d5" (UID: "b2601a6f-c711-41c0-9620-f311e6ba91d5"). InnerVolumeSpecName "kube-api-access-dgl96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:07.387142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.387107    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:07.387142 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.387143    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2601a6f-c711-41c0-9620-f311e6ba91d5-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:07.387446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.387159    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgl96\" (UniqueName: \"kubernetes.io/projected/b2601a6f-c711-41c0-9620-f311e6ba91d5-kube-api-access-dgl96\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:07.387446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.387175    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2601a6f-c711-41c0-9620-f311e6ba91d5-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:07.387446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.387343    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:07.407607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.407566    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"]
Oct 30 17:55:07.411305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.411272    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a271c-ch225"]
Oct 30 17:55:07.509110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.509075    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:07.783722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.782813    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 17:55:07.783722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:07.782877    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 17:55:08.135285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.135258    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"
Oct 30 17:55:08.135473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.135299    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh" event={"ID":"b2601a6f-c711-41c0-9620-f311e6ba91d5","Type":"ContainerDied","Data":"49dc5e97b75264749c5ec14d45a4954c607e48e2ca57ff3f86e3945326b7485f"}
Oct 30 17:55:08.135473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.135363    5534 scope.go:122] "RemoveContainer" containerID="1a44dad4a3a27ae0d24d16f50f31589429215f80d69197d6c1d6d4620b5f23e4"
Oct 30 17:55:08.200367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.200329    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"]
Oct 30 17:55:08.209472 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.209433    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a271c-64bdf7df5c-l82qh"]
Oct 30 17:55:08.236696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.236669    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:08.350908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.350862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:08.393107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.393011    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fc9f4a-37b7-4480-9244-a2d011318dab" path="/var/lib/kubelet/pods/28fc9f4a-37b7-4480-9244-a2d011318dab/volumes"
Oct 30 17:55:08.393575 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.393556    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2601a6f-c711-41c0-9620-f311e6ba91d5" path="/var/lib/kubelet/pods/b2601a6f-c711-41c0-9620-f311e6ba91d5/volumes"
Oct 30 17:55:08.607850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.607293    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"]
Oct 30 17:55:08.611594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.610429    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" containerID="cri-o://4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d" gracePeriod=30
Oct 30 17:55:08.672956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.672279    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:08.763699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.763038    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 17:55:08.763699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:08.763113    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 17:55:09.262499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:09.262459    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:55:09.277114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:09.277086    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:55:09.415931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:09.415871    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:09.780034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:09.780005    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:55:09.781285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:09.781255    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:55:09.782092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:09.782064    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 17:55:10.165175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:10.165143    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:55:11.456837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:11.456805    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:11.574837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:11.574806    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:11.691453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:11.691420    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:12.352193 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:12.352157    5534 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:55:12.352400 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:12.352224    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:55:13.296428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:13.296395    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:13.952432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:13.952406    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:55:14.016440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016407    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016478    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016515    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016555    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016596    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016679    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016789    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.016896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016859    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:14.016896 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016875    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:14.017561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016923    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.017561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.016966    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data\") pod \"992a0c91-f81c-4e53-8217-8bb93a9353b8\" (UID: \"992a0c91-f81c-4e53-8217-8bb93a9353b8\") "
Oct 30 17:55:14.017561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.017059    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:14.017561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.017222    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:14.017561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.017441    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:14.018043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.017956    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.018043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.017979    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.018043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.018029    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992a0c91-f81c-4e53-8217-8bb93a9353b8-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.018219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.018045    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.018219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.018059    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.019318 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.019295    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:14.019765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.019726    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:55:14.020032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.020004    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "kube-api-access-cs9wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:14.020435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.020414    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token" pod "992a0c91-f81c-4e53-8217-8bb93a9353b8" (UID: "992a0c91-f81c-4e53-8217-8bb93a9353b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:14.108241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.108204    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"]
Oct 30 17:55:14.108974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.108955    5534 cpu_manager.go:414] "containerMap: removing container" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.108978    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109001    5534 cpu_manager.go:414] "containerMap: removing container" podUID="79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" containerName="maas-api"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109006    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" containerName="maas-api"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109021    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b2601a6f-c711-41c0-9620-f311e6ba91d5" containerName="maas-api"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109027    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2601a6f-c711-41c0-9620-f311e6ba91d5" containerName="maas-api"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109136    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerName="istio-proxy"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109146    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b6d9c6-5bf7-446d-9261-ae9d8672aa2d" containerName="maas-api"
Oct 30 17:55:14.109287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.109155    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2601a6f-c711-41c0-9620-f311e6ba91d5" containerName="maas-api"
Oct 30 17:55:14.113136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.113115    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.119440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.119412    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.119549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.119446    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.119549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.119463    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs9wb\" (UniqueName: \"kubernetes.io/projected/992a0c91-f81c-4e53-8217-8bb93a9353b8-kube-api-access-cs9wb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.119549 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.119478    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992a0c91-f81c-4e53-8217-8bb93a9353b8-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:14.135893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.135859    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"]
Oct 30 17:55:14.207354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.207259    5534 generic.go:356] "Generic (PLEG): container finished" podID="992a0c91-f81c-4e53-8217-8bb93a9353b8" containerID="4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d" exitCode=0
Oct 30 17:55:14.207354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.207321    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"
Oct 30 17:55:14.207687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.207349    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" event={"ID":"992a0c91-f81c-4e53-8217-8bb93a9353b8","Type":"ContainerDied","Data":"4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d"}
Oct 30 17:55:14.207687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.207407    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m" event={"ID":"992a0c91-f81c-4e53-8217-8bb93a9353b8","Type":"ContainerDied","Data":"2b4168cb30ad7002c7403fdadbb65af10cac49dce51237cc15558bb4339d54fd"}
Oct 30 17:55:14.207687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.207433    5534 scope.go:122] "RemoveContainer" containerID="4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d"
Oct 30 17:55:14.219715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.219690    5534 scope.go:122] "RemoveContainer" containerID="4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d"
Oct 30 17:55:14.220040 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:14.220020    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d\": container with ID starting with 4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d not found: ID does not exist" containerID="4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d"
Oct 30 17:55:14.220108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220049    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d"} err="failed to get container status \"4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d\": rpc error: code = NotFound desc = could not find container \"4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d\": container with ID starting with 4bd86bc0ddd0bb185c0e99b5593b28fe2abe965525ebbff71441f250b458997d not found: ID does not exist"
Oct 30 17:55:14.220480 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220457    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.220521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220499    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.220604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220589    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.220950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220934    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n954t\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.221001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220960    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.221001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.220979    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.221071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.221047    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.221071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.221065    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.221238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.221221    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.259750 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.259721    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"]
Oct 30 17:55:14.264329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.264304    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-qf84m"]
Oct 30 17:55:14.322180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322142    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322241    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n954t\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322268    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322295    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322369    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322399    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322545    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.322627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322559    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.323049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322690    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.323049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.322728    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.323049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.323008    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.323172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.323013    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.323172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.323071    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.323172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.323084    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.325067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.325041    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.325184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.325117    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.345804 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.345780    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.346000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.345977    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n954t\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.371035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.371004    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992a0c91-f81c-4e53-8217-8bb93a9353b8" path="/var/lib/kubelet/pods/992a0c91-f81c-4e53-8217-8bb93a9353b8/volumes"
Oct 30 17:55:14.428258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.428224    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:14.761082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.761035    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"]
Oct 30 17:55:14.761386 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:55:14.761360    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda478a115_837d_42ec_a5d8_128378c3483d.slice/crio-078afd616c12ba925d0a0b2d6bdd095c37cb2e5fae1dc3541a38ac2c8c0f8b2a WatchSource:0}: Error finding container 078afd616c12ba925d0a0b2d6bdd095c37cb2e5fae1dc3541a38ac2c8c0f8b2a: Status 404 returned error can't find the container with id 078afd616c12ba925d0a0b2d6bdd095c37cb2e5fae1dc3541a38ac2c8c0f8b2a
Oct 30 17:55:14.764556 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.764524    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:14.764673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.764590    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:14.764673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.764621    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:14.998127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:14.998065    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:15.218514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:15.218472    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" event={"ID":"a478a115-837d-42ec-a5d8-128378c3483d","Type":"ContainerStarted","Data":"83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793"}
Oct 30 17:55:15.218746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:15.218592    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" event={"ID":"a478a115-837d-42ec-a5d8-128378c3483d","Type":"ContainerStarted","Data":"078afd616c12ba925d0a0b2d6bdd095c37cb2e5fae1dc3541a38ac2c8c0f8b2a"}
Oct 30 17:55:15.249573 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:15.249542    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:55:15.268221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:15.268158    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podStartSLOduration=1.26813872 podStartE2EDuration="1.26813872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:15.263467217 +0000 UTC m=+2521.025511026" watchObservedRunningTime="2026-10-30 17:55:15.26813872 +0000 UTC m=+2521.030182534"
Oct 30 17:55:15.428839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:15.428810    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:15.430359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:15.430330    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 17:55:15.430495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:15.430386    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 17:55:16.429429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:16.429392    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 17:55:16.429997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:16.429459    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 17:55:16.773727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:16.773244    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:17.428982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:17.428941    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 17:55:17.429281 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:17.429028    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 17:55:17.940075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:17.940043    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:18.429632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.429590    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 17:55:18.429862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.429697    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 17:55:18.522073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.522032    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:18.817271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.817186    5534 scope.go:122] "RemoveContainer" containerID="49e9eb997433c7b37990f645808952fc4d30f4a4bcb572299eae5d58a4fc0195"
Oct 30 17:55:18.828182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.828162    5534 scope.go:122] "RemoveContainer" containerID="dfffa67e179644e7e7d0b0b3fa15e15fc50853f465d7f504781ebacb3406fc2b"
Oct 30 17:55:18.838173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.838147    5534 scope.go:122] "RemoveContainer" containerID="4aa23c91887f5826098462856d60633b9bc0ef56a8bcfd09503bc2010638ed3c"
Oct 30 17:55:18.849405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.849308    5534 scope.go:122] "RemoveContainer" containerID="d9fde37ae928ea9e3abd69c0c608dfd9df1af5b0193f02e14a529f3d102d7ea4"
Oct 30 17:55:18.888253 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:18.888227    5534 scope.go:122] "RemoveContainer" containerID="c4bdedaec9badfc70007f908bcbe27140dc11b274c8a588769e36e233cdcbff0"
Oct 30 17:55:19.031561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:19.031538    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"]
Oct 30 17:55:19.045006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:19.044980    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e96046-p2v5c"]
Oct 30 17:55:19.429048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:19.429006    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 17:55:19.429343 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:19.429068    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 17:55:20.391962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:20.391922    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea224c8-d022-4f0f-84a8-900f21d00483" path="/var/lib/kubelet/pods/5ea224c8-d022-4f0f-84a8-900f21d00483/volumes"
Oct 30 17:55:20.428615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:20.428581    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 17:55:20.428800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:20.428659    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 17:55:20.621338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:20.621304    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:21.433904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:21.433873    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:21.434351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:21.434338    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:21.435184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:21.435170    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 17:55:22.445844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:22.445812    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:23.960969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:23.960932    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:24.687778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:24.687449    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:25.723201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:25.723168    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:25.845460 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:25.845422    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:26.115194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.115158    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"]
Oct 30 17:55:26.115508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.115481    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" containerID="cri-o://8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0" gracePeriod=30
Oct 30 17:55:26.135830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.135796    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:26.705545 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.705513    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"]
Oct 30 17:55:26.712208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.712183    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.763097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.763066    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"]
Oct 30 17:55:26.768348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.768319    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:55:26.768558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.768529    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:55:26.837311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.837279    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxft\" (UniqueName: \"kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.837508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.837460    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.837583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.837565    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.889636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.889595    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:26.945771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.939386    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxft\" (UniqueName: \"kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.945771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.939592    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.945771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.939712    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.945771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.940702    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.945771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.943382    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:26.964761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:26.963975    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxft\" (UniqueName: \"kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft\") pod \"maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:27.029181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:27.029152    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:27.343245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:27.343220    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"]
Oct 30 17:55:27.343812 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:55:27.343788    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276a11eb_03f6_4556_be2e_d60e226dd6ee.slice/crio-4760f0233878444984ba174aa6df102b283718058b0a27d506cc6705142e0360 WatchSource:0}: Error finding container 4760f0233878444984ba174aa6df102b283718058b0a27d506cc6705142e0360: Status 404 returned error can't find the container with id 4760f0233878444984ba174aa6df102b283718058b0a27d506cc6705142e0360
Oct 30 17:55:27.384703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:27.384669    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq" event={"ID":"276a11eb-03f6-4556-be2e-d60e226dd6ee","Type":"ContainerStarted","Data":"4760f0233878444984ba174aa6df102b283718058b0a27d506cc6705142e0360"}
Oct 30 17:55:27.772632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:27.772584    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:28.393983 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:28.393940    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq" event={"ID":"276a11eb-03f6-4556-be2e-d60e226dd6ee","Type":"ContainerStarted","Data":"b40af43b22c7735f8914b8438d8053669db8b56ee1811cf57e92a09fb566df1c"}
Oct 30 17:55:28.435846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:28.435778    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq" podStartSLOduration=2.435758862 podStartE2EDuration="2.435758862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:28.43333122 +0000 UTC m=+2534.195375034" watchObservedRunningTime="2026-10-30 17:55:28.435758862 +0000 UTC m=+2534.197802675"
Oct 30 17:55:28.921464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:28.921424    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:29.410100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:29.410064    5534 generic.go:356] "Generic (PLEG): container finished" podID="276a11eb-03f6-4556-be2e-d60e226dd6ee" containerID="b40af43b22c7735f8914b8438d8053669db8b56ee1811cf57e92a09fb566df1c" exitCode=0
Oct 30 17:55:29.410290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:29.410141    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq" event={"ID":"276a11eb-03f6-4556-be2e-d60e226dd6ee","Type":"ContainerDied","Data":"b40af43b22c7735f8914b8438d8053669db8b56ee1811cf57e92a09fb566df1c"}
Oct 30 17:55:30.612838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.612811    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:30.720861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.720757    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca\") pod \"276a11eb-03f6-4556-be2e-d60e226dd6ee\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") "
Oct 30 17:55:30.720861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.720859    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca\") pod \"276a11eb-03f6-4556-be2e-d60e226dd6ee\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") "
Oct 30 17:55:30.721246 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.721219    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca" pod "276a11eb-03f6-4556-be2e-d60e226dd6ee" (UID: "276a11eb-03f6-4556-be2e-d60e226dd6ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:30.721755 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.721722    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft\" (UniqueName: \"kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft\") pod \"276a11eb-03f6-4556-be2e-d60e226dd6ee\" (UID: \"276a11eb-03f6-4556-be2e-d60e226dd6ee\") "
Oct 30 17:55:30.722524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.722493    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:30.723693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.723457    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca" pod "276a11eb-03f6-4556-be2e-d60e226dd6ee" (UID: "276a11eb-03f6-4556-be2e-d60e226dd6ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:30.724926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.724901    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft" pod "276a11eb-03f6-4556-be2e-d60e226dd6ee" (UID: "276a11eb-03f6-4556-be2e-d60e226dd6ee"). InnerVolumeSpecName "kube-api-access-5pxft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:30.823591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.823555    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276a11eb-03f6-4556-be2e-d60e226dd6ee-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:30.823591 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.823593    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pxft\" (UniqueName: \"kubernetes.io/projected/276a11eb-03f6-4556-be2e-d60e226dd6ee-kube-api-access-5pxft\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:30.957708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.955928    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:30.975712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.972297    5534 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:55:30.975712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:30.972363    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:55:31.412830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.412804    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:55:31.431760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.431631    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"
Oct 30 17:55:31.431760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.431678    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq" event={"ID":"276a11eb-03f6-4556-be2e-d60e226dd6ee","Type":"ContainerDied","Data":"4760f0233878444984ba174aa6df102b283718058b0a27d506cc6705142e0360"}
Oct 30 17:55:31.431760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.431733    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4760f0233878444984ba174aa6df102b283718058b0a27d506cc6705142e0360"
Oct 30 17:55:31.433798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.433771    5534 generic.go:356] "Generic (PLEG): container finished" podID="42639ec9-312c-49fa-b750-3cead79a0534" containerID="8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0" exitCode=0
Oct 30 17:55:31.433920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.433821    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"
Oct 30 17:55:31.433920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.433854    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" event={"ID":"42639ec9-312c-49fa-b750-3cead79a0534","Type":"ContainerDied","Data":"8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0"}
Oct 30 17:55:31.433920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.433906    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt" event={"ID":"42639ec9-312c-49fa-b750-3cead79a0534","Type":"ContainerDied","Data":"856a6ae4f7b3de9c36c79ff5338c907bb0ac8494c06c41601c1f20e692828fff"}
Oct 30 17:55:31.434071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.433934    5534 scope.go:122] "RemoveContainer" containerID="8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0"
Oct 30 17:55:31.434512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434492    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434603 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434549    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434606    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434702    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434780    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434825    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434894    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.434995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.434952    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:31.435107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.435033    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.435500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.435110    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data\") pod \"42639ec9-312c-49fa-b750-3cead79a0534\" (UID: \"42639ec9-312c-49fa-b750-3cead79a0534\") "
Oct 30 17:55:31.435500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.435125    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:31.435943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.435875    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42639ec9-312c-49fa-b750-3cead79a0534-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.435943 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.435900    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.436926 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.436902    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:31.437028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.436934    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:31.437099 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.437087    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:31.441206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.441079    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:55:31.441206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.441177    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:55:31.443155 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.442451    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "kube-api-access-dd5qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:31.445161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.445141    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token" pod "42639ec9-312c-49fa-b750-3cead79a0534" (UID: "42639ec9-312c-49fa-b750-3cead79a0534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:31.450234 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.450209    5534 scope.go:122] "RemoveContainer" containerID="737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"
Oct 30 17:55:31.466828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.466786    5534 scope.go:122] "RemoveContainer" containerID="8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0"
Oct 30 17:55:31.467148 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:31.467123    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0\": container with ID starting with 8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0 not found: ID does not exist" containerID="8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0"
Oct 30 17:55:31.467207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.467162    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0"} err="failed to get container status \"8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0\": rpc error: code = NotFound desc = could not find container \"8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0\": container with ID starting with 8f1c110beded43d045435bf90393a2938aba30b37611ad447cd8d3b0c84e96c0 not found: ID does not exist"
Oct 30 17:55:31.467207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.467186    5534 scope.go:122] "RemoveContainer" containerID="737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"
Oct 30 17:55:31.467573 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:31.467551    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d\": container with ID starting with 737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d not found: ID does not exist" containerID="737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"
Oct 30 17:55:31.467678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.467578    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d"} err="failed to get container status \"737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d\": rpc error: code = NotFound desc = could not find container \"737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d\": container with ID starting with 737767931fa159a30ad6e8c90c4e6d79b0d814ed0eeac2b6fb384f5e6d4c414d not found: ID does not exist"
Oct 30 17:55:31.537558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537473    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd5qf\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-kube-api-access-dd5qf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.537558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537508    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.537558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537524    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.537558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537537    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.537558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537554    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42639ec9-312c-49fa-b750-3cead79a0534-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.538029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537566    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42639ec9-312c-49fa-b750-3cead79a0534-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.538029 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.537576    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42639ec9-312c-49fa-b750-3cead79a0534-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:31.789229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.789125    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"]
Oct 30 17:55:31.813717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:31.813591    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-kphkt"]
Oct 30 17:55:32.374818 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:32.374771    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42639ec9-312c-49fa-b750-3cead79a0534" path="/var/lib/kubelet/pods/42639ec9-312c-49fa-b750-3cead79a0534/volumes"
Oct 30 17:55:32.851380 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:32.851349    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:33.009905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:33.009874    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:34.610974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:34.610936    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:36.648119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:36.648080    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:37.336694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:37.331831    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:37.451354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:37.451320    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:37.931153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:37.930613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:38.046508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.046475    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:38.650867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.650835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:38.686221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.686190    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8851-6ef90b"]
Oct 30 17:55:38.686982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.686959    5534 cpu_manager.go:414] "containerMap: removing container" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy"
Oct 30 17:55:38.686982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.686977    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy"
Oct 30 17:55:38.687100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.686988    5534 cpu_manager.go:414] "containerMap: removing container" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy"
Oct 30 17:55:38.687100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.686994    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy"
Oct 30 17:55:38.687100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.687010    5534 cpu_manager.go:414] "containerMap: removing container" podUID="276a11eb-03f6-4556-be2e-d60e226dd6ee" containerName="revoke-keys"
Oct 30 17:55:38.687100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.687015    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276a11eb-03f6-4556-be2e-d60e226dd6ee" containerName="revoke-keys"
Oct 30 17:55:38.687242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.687123    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy"
Oct 30 17:55:38.687242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.687141    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="42639ec9-312c-49fa-b750-3cead79a0534" containerName="istio-proxy"
Oct 30 17:55:38.687242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.687150    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="276a11eb-03f6-4556-be2e-d60e226dd6ee" containerName="revoke-keys"
Oct 30 17:55:38.693014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.692994    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:38.707761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.707725    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8851-6ef90b"]
Oct 30 17:55:38.718014 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.717986    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5c6gd\""
Oct 30 17:55:38.785081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.784604    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:38.833373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.832475    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lm2\" (UniqueName: \"kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2\") pod \"test-curl-8851-6ef90b\" (UID: \"1cfe6892-c8f2-46c1-978d-fa6550fb9424\") " pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:38.934269 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.934189    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lm2\" (UniqueName: \"kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2\") pod \"test-curl-8851-6ef90b\" (UID: \"1cfe6892-c8f2-46c1-978d-fa6550fb9424\") " pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:38.983670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:38.982081    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lm2\" (UniqueName: \"kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2\") pod \"test-curl-8851-6ef90b\" (UID: \"1cfe6892-c8f2-46c1-978d-fa6550fb9424\") " pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:39.020487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:39.020454    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:39.392662 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:39.390969    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8851-6ef90b"]
Oct 30 17:55:39.539240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:39.539211    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:39.546293 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:39.546258    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8851-6ef90b" event={"ID":"1cfe6892-c8f2-46c1-978d-fa6550fb9424","Type":"ContainerStarted","Data":"3b01cc813ab83ce203c4102015333926dde6486d215616bb7886b7eee4ecccc1"}
Oct 30 17:55:40.603228 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:40.603191    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:42.339348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:42.339317    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:42.616564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:42.616531    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8851-6ef90b" event={"ID":"1cfe6892-c8f2-46c1-978d-fa6550fb9424","Type":"ContainerStarted","Data":"3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7"}
Oct 30 17:55:42.655065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:42.655014    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8851-6ef90b" podStartSLOduration=1.9528917730000002 podStartE2EDuration="4.655000589s" totalImagesPullingTime="2.702108816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:42.653523349 +0000 UTC m=+2548.415567162" watchObservedRunningTime="2026-10-30 17:55:42.655000589 +0000 UTC m=+2548.417044401"
Oct 30 17:55:43.681945 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:43.681912    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8851-6ef90b"]
Oct 30 17:55:43.686522 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:43.686494    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8851-6ef90b"]
Oct 30 17:55:44.002351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.002235    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"]
Oct 30 17:55:44.003365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.003343    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" containerName="test-curl-8851-6ef90b"
Oct 30 17:55:44.003443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.003370    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" containerName="test-curl-8851-6ef90b"
Oct 30 17:55:44.003614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.003599    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" containerName="test-curl-8851-6ef90b"
Oct 30 17:55:44.031531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.031500    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"]
Oct 30 17:55:44.031727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.031570    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.058405 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.058374    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:55:44.058568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.058431    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:55:44.146427 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.146395    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.146653 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.146626    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.146726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.146683    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mntw\" (UniqueName: \"kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.206998 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.206966    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:44.253097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.252993    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.253097 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.253071    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mntw\" (UniqueName: \"kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.253354 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.253286    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.255013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.254524    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.256955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.256867    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.289858 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.289824    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mntw\" (UniqueName: \"kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw\") pod \"maas-api-revoke-sub-6a204bd7-rk2vv\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.368225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.368184    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:44.633070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.633030    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8851-6ef90b" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" containerName="test-curl-8851-6ef90b" containerID="cri-o://3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7" gracePeriod=2
Oct 30 17:55:44.701024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:44.700996    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"]
Oct 30 17:55:44.703890 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:55:44.703860    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631a6569_80ae_41bc_991f_11d7b61c4cb2.slice/crio-24df2ea71fd217ac091f06c8aa7dfe4511f7d64776657f8563e64487d766a426 WatchSource:0}: Error finding container 24df2ea71fd217ac091f06c8aa7dfe4511f7d64776657f8563e64487d766a426: Status 404 returned error can't find the container with id 24df2ea71fd217ac091f06c8aa7dfe4511f7d64776657f8563e64487d766a426
Oct 30 17:55:45.245760 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:45.245677    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:55:45.642503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.642472    5534 generic.go:356] "Generic (PLEG): container finished" podID="88e416bc-4744-4354-b782-9ae4e6c64217" containerID="2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a" exitCode=0
Oct 30 17:55:45.642686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.642542    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" event={"ID":"88e416bc-4744-4354-b782-9ae4e6c64217","Type":"ContainerDied","Data":"2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"}
Oct 30 17:55:45.643106 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.643086    5534 scope.go:122] "RemoveContainer" containerID="2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"
Oct 30 17:55:45.644598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.644576    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv" event={"ID":"631a6569-80ae-41bc-991f-11d7b61c4cb2","Type":"ContainerStarted","Data":"bb6db243a709860c805fd054b9fbc525005dd9826a0d70d46f8079301fa05208"}
Oct 30 17:55:45.644729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.644625    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv" event={"ID":"631a6569-80ae-41bc-991f-11d7b61c4cb2","Type":"ContainerStarted","Data":"24df2ea71fd217ac091f06c8aa7dfe4511f7d64776657f8563e64487d766a426"}
Oct 30 17:55:45.645240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.645211    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:45.645311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.645296    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:45.645360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:45.645345    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:46.028531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.028419    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv" podStartSLOduration=3.028402782 podStartE2EDuration="3.028402782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:45.833430777 +0000 UTC m=+2551.595474591" watchObservedRunningTime="2026-10-30 17:55:46.028402782 +0000 UTC m=+2551.790446596"
Oct 30 17:55:46.032345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.032318    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"]
Oct 30 17:55:46.038440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.038408    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc4b213-849zb"]
Oct 30 17:55:46.050766 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.050739    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:46.339201 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.339118    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:55:46.373373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.373329    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ed7c40-4e5f-4748-8735-d163d6360fcb" path="/var/lib/kubelet/pods/88ed7c40-4e5f-4748-8735-d163d6360fcb/volumes"
Oct 30 17:55:46.688669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.687447    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" event={"ID":"88e416bc-4744-4354-b782-9ae4e6c64217","Type":"ContainerStarted","Data":"1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97"}
Oct 30 17:55:46.693023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.692987    5534 generic.go:356] "Generic (PLEG): container finished" podID="631a6569-80ae-41bc-991f-11d7b61c4cb2" containerID="bb6db243a709860c805fd054b9fbc525005dd9826a0d70d46f8079301fa05208" exitCode=0
Oct 30 17:55:46.693232 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.693162    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv" event={"ID":"631a6569-80ae-41bc-991f-11d7b61c4cb2","Type":"ContainerDied","Data":"bb6db243a709860c805fd054b9fbc525005dd9826a0d70d46f8079301fa05208"}
Oct 30 17:55:46.935954 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:46.935931    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:47.000828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.000790    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2\" (UniqueName: \"kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2\") pod \"1cfe6892-c8f2-46c1-978d-fa6550fb9424\" (UID: \"1cfe6892-c8f2-46c1-978d-fa6550fb9424\") "
Oct 30 17:55:47.003523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.003495    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2" pod "1cfe6892-c8f2-46c1-978d-fa6550fb9424" (UID: "1cfe6892-c8f2-46c1-978d-fa6550fb9424"). InnerVolumeSpecName "kube-api-access-b8lm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:47.103081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.103047    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lm2\" (UniqueName: \"kubernetes.io/projected/1cfe6892-c8f2-46c1-978d-fa6550fb9424-kube-api-access-b8lm2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:47.344231 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.344204    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:55:47.345327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.345303    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 17:55:47.345458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.345349    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 17:55:47.705825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.705713    5534 generic.go:356] "Generic (PLEG): container finished" podID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" containerID="3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7" exitCode=137
Oct 30 17:55:47.706737 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.706715    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:47.716730 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.713804    5534 scope.go:122] "RemoveContainer" containerID="3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7"
Oct 30 17:55:47.731406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.731377    5534 scope.go:122] "RemoveContainer" containerID="3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7"
Oct 30 17:55:47.731816 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:47.731789    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7\": container with ID starting with 3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7 not found: ID does not exist" containerID="3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7"
Oct 30 17:55:47.731919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.731823    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7"} err="failed to get container status \"3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7\": rpc error: code = NotFound desc = could not find container \"3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7\": container with ID starting with 3a04cef43690a736323c196a0f31ee1d6fe860f04dfa8985ae41ddaa7c3d1de7 not found: ID does not exist"
Oct 30 17:55:47.732887 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:47.732736    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8851-6ef90b\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:47.990055 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:47.989985    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:48.003557 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:48.003522    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8851-6ef90b\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" pod="opendatahub/test-curl-8851-6ef90b"
Oct 30 17:55:48.022048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.022027    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca\") pod \"631a6569-80ae-41bc-991f-11d7b61c4cb2\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") "
Oct 30 17:55:48.022235 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.022223    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw\" (UniqueName: \"kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw\") pod \"631a6569-80ae-41bc-991f-11d7b61c4cb2\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") "
Oct 30 17:55:48.022286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.022259    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca\") pod \"631a6569-80ae-41bc-991f-11d7b61c4cb2\" (UID: \"631a6569-80ae-41bc-991f-11d7b61c4cb2\") "
Oct 30 17:55:48.022351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.022326    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca" pod "631a6569-80ae-41bc-991f-11d7b61c4cb2" (UID: "631a6569-80ae-41bc-991f-11d7b61c4cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:48.022630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.022606    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:48.024394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.024374    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw" pod "631a6569-80ae-41bc-991f-11d7b61c4cb2" (UID: "631a6569-80ae-41bc-991f-11d7b61c4cb2"). InnerVolumeSpecName "kube-api-access-6mntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:48.024552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.024532    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca" pod "631a6569-80ae-41bc-991f-11d7b61c4cb2" (UID: "631a6569-80ae-41bc-991f-11d7b61c4cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:48.044360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.044335    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:48.124285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.124251    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mntw\" (UniqueName: \"kubernetes.io/projected/631a6569-80ae-41bc-991f-11d7b61c4cb2-kube-api-access-6mntw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:48.124285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.124281    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/631a6569-80ae-41bc-991f-11d7b61c4cb2-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:48.344631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.344604    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 17:55:48.344814 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.344686    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 17:55:48.389879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.389837    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfe6892-c8f2-46c1-978d-fa6550fb9424" path="/var/lib/kubelet/pods/1cfe6892-c8f2-46c1-978d-fa6550fb9424/volumes"
Oct 30 17:55:48.496173 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:55:48.496142    5534 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631a6569_80ae_41bc_991f_11d7b61c4cb2.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 17:55:48.716769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.716686    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"
Oct 30 17:55:48.716769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.716698    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv" event={"ID":"631a6569-80ae-41bc-991f-11d7b61c4cb2","Type":"ContainerDied","Data":"24df2ea71fd217ac091f06c8aa7dfe4511f7d64776657f8563e64487d766a426"}
Oct 30 17:55:48.716769 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:48.716754    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24df2ea71fd217ac091f06c8aa7dfe4511f7d64776657f8563e64487d766a426"
Oct 30 17:55:49.343852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:49.343818    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 17:55:49.344249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:49.343874    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 17:55:50.010612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:50.010567    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:50.344021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:50.343988    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 17:55:50.344454 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:50.344054    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 17:55:51.343948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:51.343808    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 17:55:51.343948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:51.343867    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 17:55:51.712010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:51.711928    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:52.348898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:52.348867    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:55:52.349370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:52.349167    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:55:52.349962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:52.349946    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 17:55:53.474974 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.474933    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"]
Oct 30 17:55:53.476695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.476661    5534 cpu_manager.go:414] "containerMap: removing container" podUID="631a6569-80ae-41bc-991f-11d7b61c4cb2" containerName="revoke-keys"
Oct 30 17:55:53.476695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.476690    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631a6569-80ae-41bc-991f-11d7b61c4cb2" containerName="revoke-keys"
Oct 30 17:55:53.476917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.476852    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="631a6569-80ae-41bc-991f-11d7b61c4cb2" containerName="revoke-keys"
Oct 30 17:55:53.483796 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.483772    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.498868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.498844    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-4dcdbb-openshift-default-dockercfg-cq2mc\""
Oct 30 17:55:53.514801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.514778    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"]
Oct 30 17:55:53.532311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532280    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532316    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532437    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532523    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfnd\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532576    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532612    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532729    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532838    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.532931 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.532901    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.552512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.552490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:53.634299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634259    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634390    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634417    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634482    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634499    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfnd\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634581    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634634    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634720    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634806    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634929    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.634991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634975    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.635158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.634993    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.635158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.635147    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.635485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.635463    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.637189 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.637169    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.637362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.637337    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.658207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.658176    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfnd\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.658768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.658735    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:53.800304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:53.800221    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:54.131842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.131811    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"]
Oct 30 17:55:54.132811 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:55:54.132783    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98ef006_a19a_49c4_b9a3_6f9a327d1b1a.slice/crio-e2ed22870d495baa8db5b45fbe95a136bce381d5e1aca8e913b869aaf63fe3e7 WatchSource:0}: Error finding container e2ed22870d495baa8db5b45fbe95a136bce381d5e1aca8e913b869aaf63fe3e7: Status 404 returned error can't find the container with id e2ed22870d495baa8db5b45fbe95a136bce381d5e1aca8e913b869aaf63fe3e7
Oct 30 17:55:54.135611 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.135578    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:54.135749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.135680    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:54.135749 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.135729    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:55:54.320048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.320013    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:54.444952 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.444852    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"]
Oct 30 17:55:54.452528 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.452502    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.468478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.467434    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:55:54.468478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.467521    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:55:54.471987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.471959    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"]
Oct 30 17:55:54.583359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.583223    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:54.653962 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.653924    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsvp\" (UniqueName: \"kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.654168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.654019    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.654168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.654052    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.756081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.755981    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.756081 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.756035    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.756361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.756350    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsvp\" (UniqueName: \"kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.756853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.756825    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.758901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.758871    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.765982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.765946    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" event={"ID":"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a","Type":"ContainerStarted","Data":"5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335"}
Oct 30 17:55:54.766124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.766009    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" event={"ID":"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a","Type":"ContainerStarted","Data":"e2ed22870d495baa8db5b45fbe95a136bce381d5e1aca8e913b869aaf63fe3e7"}
Oct 30 17:55:54.779829 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.779802    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsvp\" (UniqueName: \"kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp\") pod \"maas-api-revoke-sub-e3719eb9-k8bcd\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:54.800604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.800580    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:55:54.815602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:54.815555    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podStartSLOduration=1.815541508 podStartE2EDuration="1.815541508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:54.814632725 +0000 UTC m=+2560.576676539" watchObservedRunningTime="2026-10-30 17:55:54.815541508 +0000 UTC m=+2560.577585324"
Oct 30 17:55:55.030598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.030518    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"]
Oct 30 17:55:55.033708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.033684    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-938023d5-pljm2"]
Oct 30 17:55:55.079666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.078793    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:55.214861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.214826    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:55.396503 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.396474    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"]
Oct 30 17:55:55.397620 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:55:55.397597    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10f806f_d450_4cd1_ab0c_3f8ef9713b49.slice/crio-1cfd33cc6863cc0c4402db0d52fd52e66e2c2ac39663629191c7b61a0d7eb7c3 WatchSource:0}: Error finding container 1cfd33cc6863cc0c4402db0d52fd52e66e2c2ac39663629191c7b61a0d7eb7c3: Status 404 returned error can't find the container with id 1cfd33cc6863cc0c4402db0d52fd52e66e2c2ac39663629191c7b61a0d7eb7c3
Oct 30 17:55:55.764667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.752749    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:55.805803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.805464    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 17:55:55.805803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.805523    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 17:55:55.822531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.822490    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd" event={"ID":"a10f806f-d450-4cd1-ab0c-3f8ef9713b49","Type":"ContainerStarted","Data":"1cfd33cc6863cc0c4402db0d52fd52e66e2c2ac39663629191c7b61a0d7eb7c3"}
Oct 30 17:55:55.883342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.883290    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:55.935465 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.935429    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"]
Oct 30 17:55:55.940765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.940741    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c596f1-4xnlq"]
Oct 30 17:55:55.997552 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:55.997463    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:56.392620 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.392578    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102a1711-2402-4fb0-81ad-07eed42da980" path="/var/lib/kubelet/pods/102a1711-2402-4fb0-81ad-07eed42da980/volumes"
Oct 30 17:55:56.393175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.393154    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276a11eb-03f6-4556-be2e-d60e226dd6ee" path="/var/lib/kubelet/pods/276a11eb-03f6-4556-be2e-d60e226dd6ee/volumes"
Oct 30 17:55:56.448652 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.448621    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:56.800600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.800569    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 17:55:56.801091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.800626    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 17:55:56.831036 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.830998    5534 generic.go:356] "Generic (PLEG): container finished" podID="a10f806f-d450-4cd1-ab0c-3f8ef9713b49" containerID="cd7d432784c41c691349ac3c5f5caebdc52f2d92f58761d8049ec2d7e5490049" exitCode=0
Oct 30 17:55:56.831212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:56.831140    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd" event={"ID":"a10f806f-d450-4cd1-ab0c-3f8ef9713b49","Type":"ContainerDied","Data":"cd7d432784c41c691349ac3c5f5caebdc52f2d92f58761d8049ec2d7e5490049"}
Oct 30 17:55:57.800836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:57.800801    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 17:55:57.801388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:57.800863    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 17:55:58.034181 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.034148    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:58.115710 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.115681    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:58.207795 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.207757    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp\" (UniqueName: \"kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp\") pod \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") "
Oct 30 17:55:58.208609 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.208582    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca\") pod \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") "
Oct 30 17:55:58.209030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.209006    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca\") pod \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\" (UID: \"a10f806f-d450-4cd1-ab0c-3f8ef9713b49\") "
Oct 30 17:55:58.210095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.209150    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca" pod "a10f806f-d450-4cd1-ab0c-3f8ef9713b49" (UID: "a10f806f-d450-4cd1-ab0c-3f8ef9713b49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:55:58.215107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.215067    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp" pod "a10f806f-d450-4cd1-ab0c-3f8ef9713b49" (UID: "a10f806f-d450-4cd1-ab0c-3f8ef9713b49"). InnerVolumeSpecName "kube-api-access-mlsvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:55:58.216622 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.216288    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca" pod "a10f806f-d450-4cd1-ab0c-3f8ef9713b49" (UID: "a10f806f-d450-4cd1-ab0c-3f8ef9713b49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:55:58.322881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.318441    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:58.322881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.318479    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlsvp\" (UniqueName: \"kubernetes.io/projected/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-kube-api-access-mlsvp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:58.322881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.318495    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10f806f-d450-4cd1-ab0c-3f8ef9713b49-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:55:58.808692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.807052    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 17:55:58.808692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.807113    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 17:55:58.881569 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.881531    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd" event={"ID":"a10f806f-d450-4cd1-ab0c-3f8ef9713b49","Type":"ContainerDied","Data":"1cfd33cc6863cc0c4402db0d52fd52e66e2c2ac39663629191c7b61a0d7eb7c3"}
Oct 30 17:55:58.881864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.881848    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cfd33cc6863cc0c4402db0d52fd52e66e2c2ac39663629191c7b61a0d7eb7c3"
Oct 30 17:55:58.882063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:58.882050    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"
Oct 30 17:55:59.213897 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:59.213207    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:55:59.800950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:59.800910    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 17:55:59.801145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:55:59.800981    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 17:56:00.238787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.238720    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:00.461763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.461722    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"]
Oct 30 17:56:00.463032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.463002    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a10f806f-d450-4cd1-ab0c-3f8ef9713b49" containerName="revoke-keys"
Oct 30 17:56:00.463032 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.463027    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10f806f-d450-4cd1-ab0c-3f8ef9713b49" containerName="revoke-keys"
Oct 30 17:56:00.463260 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.463245    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10f806f-d450-4cd1-ab0c-3f8ef9713b49" containerName="revoke-keys"
Oct 30 17:56:00.467115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.467096    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.490904 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.490818    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"]
Oct 30 17:56:00.496003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.495980    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-d53ec1-openshift-default-dockercfg-2q58v\""
Oct 30 17:56:00.522912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.522872    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmrld\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.522923    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523042    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523120 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523090    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523176    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523316    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523358    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523396    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.523513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.523471    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.625578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625547    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.625826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625585    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.625826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625616    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.625826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625692    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.625826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625784    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmrld\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.625826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625800    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625860    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625892    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.625943    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626166 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.626036    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.626328    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626861 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.626833    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.626992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.626860    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.627056 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.626983    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.629007 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.628983    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.629593 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.629571    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.649836 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.649800    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmrld\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.650001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.649971    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.783810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.783695    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:00.801199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.801168    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 17:56:00.801334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:00.801232    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 17:56:01.123560 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.123524    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"]
Oct 30 17:56:01.124233 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:56:01.124201    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6135862_99d6_40be_af38_b0ee8856ccf7.slice/crio-8409c59df91bcc61e6479fb2d72f3f4828d6d0f0db8670855710959d6805c0b0 WatchSource:0}: Error finding container 8409c59df91bcc61e6479fb2d72f3f4828d6d0f0db8670855710959d6805c0b0: Status 404 returned error can't find the container with id 8409c59df91bcc61e6479fb2d72f3f4828d6d0f0db8670855710959d6805c0b0
Oct 30 17:56:01.127361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.127322    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:01.127461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.127405    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:01.127461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.127451    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:01.806393 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.806353    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:56:01.806919 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.806872    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:56:01.807787 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.807763    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 17:56:01.921798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.921707    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" event={"ID":"b6135862-99d6-40be-af38-b0ee8856ccf7","Type":"ContainerStarted","Data":"5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1"}
Oct 30 17:56:01.921798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.921758    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" event={"ID":"b6135862-99d6-40be-af38-b0ee8856ccf7","Type":"ContainerStarted","Data":"8409c59df91bcc61e6479fb2d72f3f4828d6d0f0db8670855710959d6805c0b0"}
Oct 30 17:56:01.995667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:01.994790    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podStartSLOduration=1.994771581 podStartE2EDuration="1.994771581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:01.970209133 +0000 UTC m=+2567.732252980" watchObservedRunningTime="2026-10-30 17:56:01.994771581 +0000 UTC m=+2567.756815466"
Oct 30 17:56:02.022039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:02.022003    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:02.784304 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:02.784269    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:02.786225 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:02.786194    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 17:56:02.786385 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:02.786255    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 17:56:03.104087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.104049    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:03.242513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.242479    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:03.417342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.417242    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"]
Oct 30 17:56:03.417597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.417565    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy" containerID="cri-o://c30c7caeb8f0c695b533ec7d6eba52347bc75216c508ec9a1a50d239acb3d895" gracePeriod=30
Oct 30 17:56:03.472669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.464792    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:03.786727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.786205    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 17:56:03.786727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:03.786308    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 17:56:04.223092 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:04.223063    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:04.347149 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:04.347117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:04.791289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:04.791247    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 17:56:04.791509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:04.791332    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 17:56:04.803306 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:04.803270    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:04.912751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:04.912710    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:05.784289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:05.784242    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 17:56:05.784760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:05.784313    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 17:56:06.021827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.021786    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:06.701464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.701434    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"]
Oct 30 17:56:06.711546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.711511    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.738110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.736135    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"]
Oct 30 17:56:06.739915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.739755    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"default-dockercfg-6qvqm\""
Oct 30 17:56:06.739915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.739852    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"sub-test-model-23c1e8-kserve-self-signed-certs\""
Oct 30 17:56:06.743744 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.743691    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"openshift-service-ca.crt\""
Oct 30 17:56:06.744531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.744505    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"kube-root-ca.crt\""
Oct 30 17:56:06.785524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.784023    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:06.785524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.785102    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 17:56:06.785524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.785150    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 17:56:06.884992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.884950    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.885194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.885014    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.885194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.885063    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.885194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.885156    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.885368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.885208    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.885426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.885407    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdv6\" (UniqueName: \"kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.989961    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.990101    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdv6\" (UniqueName: \"kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.990237    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.990286    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.990311    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.990372    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.990877    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:06.991720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:06.991118    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:07.004243 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.003830    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:07.064666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.058734    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:07.064666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.062266    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdv6\" (UniqueName: \"kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:07.064666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.062416    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm\") pod \"sub-test-model-23c1e8-kserve-58c749c88f-8dwb5\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:07.346590 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.346559    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:07.680244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.680214    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"]
Oct 30 17:56:07.680408 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:56:07.680349    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aa60a85_3c0b_4bfc_a756_f6f0cd83d409.slice/crio-8f83219461e2af03e55fc341e5352e685b94b652a9f5da2a499cffcf9aa1159a WatchSource:0}: Error finding container 8f83219461e2af03e55fc341e5352e685b94b652a9f5da2a499cffcf9aa1159a: Status 404 returned error can't find the container with id 8f83219461e2af03e55fc341e5352e685b94b652a9f5da2a499cffcf9aa1159a
Oct 30 17:56:07.790064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.790030    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:07.790848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.790818    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:07.791469 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.791443    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 17:56:07.861676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:07.857186    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:08.067789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:08.067697    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5" event={"ID":"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409","Type":"ContainerStarted","Data":"826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d"}
Oct 30 17:56:08.067789 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:08.067744    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5" event={"ID":"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409","Type":"ContainerStarted","Data":"8f83219461e2af03e55fc341e5352e685b94b652a9f5da2a499cffcf9aa1159a"}
Oct 30 17:56:08.110250 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:08.110197    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5" podStartSLOduration=2.110182756 podStartE2EDuration="2.110182756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:08.107700487 +0000 UTC m=+2573.869744301" watchObservedRunningTime="2026-10-30 17:56:08.110182756 +0000 UTC m=+2573.872226569"
Oct 30 17:56:08.347249 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:08.347216    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:08.993535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:08.993495    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:09.083353 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.083313    5534 generic.go:356] "Generic (PLEG): container finished" podID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerID="c30c7caeb8f0c695b533ec7d6eba52347bc75216c508ec9a1a50d239acb3d895" exitCode=0
Oct 30 17:56:09.084627 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.084544    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" event={"ID":"d619970a-5638-4ae2-b0a9-c647e21f496a","Type":"ContainerDied","Data":"c30c7caeb8f0c695b533ec7d6eba52347bc75216c508ec9a1a50d239acb3d895"}
Oct 30 17:56:09.110462 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.110435    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:09.132079 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.132052    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:56:09.230196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230103    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230166    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230199    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230273    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230345    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230378    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230448    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230528    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230610    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket\") pod \"d619970a-5638-4ae2-b0a9-c647e21f496a\" (UID: \"d619970a-5638-4ae2-b0a9-c647e21f496a\") "
Oct 30 17:56:09.230935 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230902    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:56:09.231021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.230997    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:56:09.231220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.231198    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:56:09.231459 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.231433    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:56:09.231708 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.231484    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:56:09.232071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.232048    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d619970a-5638-4ae2-b0a9-c647e21f496a-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.232165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.232078    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.232165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.232093    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.232165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.232107    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.232165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.232121    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.234144 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.233164    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:56:09.234572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.234547    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:56:09.234831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.234802    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:56:09.235111 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.235088    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45" pod "d619970a-5638-4ae2-b0a9-c647e21f496a" (UID: "d619970a-5638-4ae2-b0a9-c647e21f496a"). InnerVolumeSpecName "kube-api-access-mdx45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:56:09.334091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.334037    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdx45\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-kube-api-access-mdx45\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.334091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.334087    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.334091 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.334101    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.334489 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.334113    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d619970a-5638-4ae2-b0a9-c647e21f496a-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:56:09.807806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.807764    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:09.874060 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.874022    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"]
Oct 30 17:56:09.875839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.875798    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy"
Oct 30 17:56:09.875839 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.875830    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy"
Oct 30 17:56:09.876065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.876020    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" containerName="istio-proxy"
Oct 30 17:56:09.880594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.880574    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.900390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.900361    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"]
Oct 30 17:56:09.907999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.907965    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a17354-openshift-default-dockercfg-gqtvv\""
Oct 30 17:56:09.942585 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.942548    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.942849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.942631    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.942849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.942678    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.942849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.942731    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.942849 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.942774    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.943774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.942852    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.943774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.943096    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.943774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.943177    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5gb\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:09.943774 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:09.943237    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.044431 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.044393    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.044937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.044481    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.044937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.044511    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.044937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.044629    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.044937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.044739    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045022    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045154 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045134    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045245 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045206    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045321 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045300    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045357    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045452 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045431    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5gb\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045486    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045569 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045527    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.045850 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.045821    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.047140 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.047117    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.048837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.048816    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.080447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.080364    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.090028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.089999    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5gb\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.093308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.093283    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"
Oct 30 17:56:10.093448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.093307    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg" event={"ID":"d619970a-5638-4ae2-b0a9-c647e21f496a","Type":"ContainerDied","Data":"8d4bd37b1fa72648b57c7fb1e7d81d8c999877d42128dfe60a813e9983d04823"}
Oct 30 17:56:10.093448 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.093379    5534 scope.go:122] "RemoveContainer" containerID="c30c7caeb8f0c695b533ec7d6eba52347bc75216c508ec9a1a50d239acb3d895"
Oct 30 17:56:10.191681 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.191627    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"]
Oct 30 17:56:10.200109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.200080    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-nrtvg"]
Oct 30 17:56:10.200387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.200364    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:10.373761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.373730    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d619970a-5638-4ae2-b0a9-c647e21f496a" path="/var/lib/kubelet/pods/d619970a-5638-4ae2-b0a9-c647e21f496a/volumes"
Oct 30 17:56:10.531759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.531721    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"]
Oct 30 17:56:10.531892 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:56:10.531858    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0178d6_695e_4378_a7d3_a046df89a935.slice/crio-7702fbe4235a89c939204752ab34346d242fa5bf9410179d4ca38b8b7fb9f068 WatchSource:0}: Error finding container 7702fbe4235a89c939204752ab34346d242fa5bf9410179d4ca38b8b7fb9f068: Status 404 returned error can't find the container with id 7702fbe4235a89c939204752ab34346d242fa5bf9410179d4ca38b8b7fb9f068
Oct 30 17:56:10.534914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.534881    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:10.535037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.534948    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:10.535037 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.534977    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:10.757265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.757178    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:10.886938 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:10.886902    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:11.113991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.113951    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" event={"ID":"ac0178d6-695e-4378-a7d3-a046df89a935","Type":"ContainerStarted","Data":"8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b"}
Oct 30 17:56:11.114509 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.114082    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" event={"ID":"ac0178d6-695e-4378-a7d3-a046df89a935","Type":"ContainerStarted","Data":"7702fbe4235a89c939204752ab34346d242fa5bf9410179d4ca38b8b7fb9f068"}
Oct 30 17:56:11.162027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.161960    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podStartSLOduration=2.161938301 podStartE2EDuration="2.161938301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:11.157865393 +0000 UTC m=+2576.919909228" watchObservedRunningTime="2026-10-30 17:56:11.161938301 +0000 UTC m=+2576.923982121"
Oct 30 17:56:11.201239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.201208    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:11.203185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.203144    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 17:56:11.203350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.203208    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 17:56:11.818417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:11.818387    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:12.200911 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:12.200878    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 17:56:12.201326 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:12.200938    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 17:56:13.200853 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:13.200808    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 17:56:13.201027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:13.200875    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 17:56:13.616809 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:13.616763    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:14.200915 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:14.200877    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 17:56:14.201125 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:14.200957    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 17:56:15.201673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:15.200782    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 17:56:15.201673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:15.200853    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 17:56:15.246094 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:56:15.246061    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:56:15.497727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:15.496259    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:16.200898 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:16.200852    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 17:56:16.201101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:16.200913    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 17:56:17.243688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:17.243633    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:17.329760 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:17.329712    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:17.330927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:17.330904    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:17.331720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:17.331699    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 17:56:17.347289 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:17.347271    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:17.360080 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:17.360063    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:18.211179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:18.211152    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 17:56:19.046890 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.044696    5534 scope.go:122] "RemoveContainer" containerID="ffbc5f0f6df80b48fc46f099203976448a34b5f85e9382d08390c7bf8f754aa6"
Oct 30 17:56:19.065561 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.065536    5534 scope.go:122] "RemoveContainer" containerID="b500ba887bad3feca63a91ba58ecf93ed23beaa6557c16e7ad3e140de3398f0e"
Oct 30 17:56:19.081615 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.081586    5534 scope.go:122] "RemoveContainer" containerID="4849b47599ce15dd05f5bf8293b7f240dd412a3a23125a981ab82bb38c149211"
Oct 30 17:56:19.094213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.094184    5534 scope.go:122] "RemoveContainer" containerID="67d4c7ae0005eb9629dc523ccae492bffd625ce2063e512104041257d97b8dfd"
Oct 30 17:56:19.108432 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.108408    5534 scope.go:122] "RemoveContainer" containerID="9484688bbb0b136d4aeb36ae198759f9e9e5cef66f8ecb48a34d6cd5014ab730"
Oct 30 17:56:19.112899 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.112859    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:19.389948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:19.389915    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:20.907058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:20.907027    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:22.490793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:22.490758    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:22.636828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:22.636793    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:24.389379 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:24.389329    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:26.189172 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:26.189142    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:27.873791 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:27.873760    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:29.836430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:29.836382    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:30.046481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:30.046450    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"]
Oct 30 17:56:30.064577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:30.064528    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65035508-xzxt4"]
Oct 30 17:56:30.396301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:30.396262    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181ba2cf-0774-4ab2-abec-60d6bbdf65a9" path="/var/lib/kubelet/pods/181ba2cf-0774-4ab2-abec-60d6bbdf65a9/volumes"
Oct 30 17:56:31.804944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:31.804912    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:33.143090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:33.143056    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:33.555533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:33.555436    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:35.079259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:35.079221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:35.240008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:35.239978    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:37.114921 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:37.114884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:38.897291 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:38.897257    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:40.571745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:40.571706    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:42.415846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:42.415807    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:43.576851 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:43.576813    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:44.054202 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.054167    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:44.251367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.251321    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:44.368150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.368119    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:44.833329 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.833233    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"]
Oct 30 17:56:44.927712 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.914607    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:44.929169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.929134    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"]
Oct 30 17:56:44.929331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.929297    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:44.944369 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:44.944341    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-21870f-openshift-default-dockercfg-nrqzb\""
Oct 30 17:56:45.031398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031362    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031412    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031448    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031595    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031745 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031670    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031796    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4p6\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031820    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031847    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.031924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.031874    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.132985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.132947    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.132998    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133136    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133210 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133185    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133223    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133378    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4p6\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133415 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133405    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133433    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133377    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133471    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133491    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.133698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133519    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.134001 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.133849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.134063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.134051    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.135734 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.135707    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.136082 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.136061    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.156756 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.156715    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4p6\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.156906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.156808    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token\") pod \"e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") " pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.245840 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:56:45.245806    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:56:45.262344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.262317    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:45.263287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.263260    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:45.584965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.584926    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"]
Oct 30 17:56:45.585660 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:56:45.585620    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3612c751_1efc_4498_b1aa_2b8e47e45e8c.slice/crio-9b97d6f02a58c9efaf6e11154ccbb6ed23855a02a45e8b57d9ee35dc66e06612 WatchSource:0}: Error finding container 9b97d6f02a58c9efaf6e11154ccbb6ed23855a02a45e8b57d9ee35dc66e06612: Status 404 returned error can't find the container with id 9b97d6f02a58c9efaf6e11154ccbb6ed23855a02a45e8b57d9ee35dc66e06612
Oct 30 17:56:45.588351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.588321    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:45.588428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.588382    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:45.588428 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:45.588411    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:56:46.011449 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:46.011419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:46.477966 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:46.477929    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" event={"ID":"3612c751-1efc-4498-b1aa-2b8e47e45e8c","Type":"ContainerStarted","Data":"e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"}
Oct 30 17:56:46.478158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:46.477989    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" event={"ID":"3612c751-1efc-4498-b1aa-2b8e47e45e8c","Type":"ContainerStarted","Data":"9b97d6f02a58c9efaf6e11154ccbb6ed23855a02a45e8b57d9ee35dc66e06612"}
Oct 30 17:56:46.523637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:46.523584    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podStartSLOduration=2.52357063 podStartE2EDuration="2.52357063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:46.521794097 +0000 UTC m=+2612.283837913" watchObservedRunningTime="2026-10-30 17:56:46.52357063 +0000 UTC m=+2612.285614442"
Oct 30 17:56:47.264011 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:47.263974    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:47.271583 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:47.271553    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:47.489826 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:47.489796    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:47.490982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:47.490957    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:56:47.812636 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:47.812584    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:48.886359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:48.886329    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:49.001765 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:49.001731    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:49.717089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:49.717057    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:50.601404 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:50.601363    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:50.720806 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:50.720770    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:51.543797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:51.543770    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:53.425026 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:53.424985    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:54.762034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:54.761931    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:54.870510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:54.870480    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:55.029892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:55.029803    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:55.521894 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:55.521853    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:56.710284 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:56.710253    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:56:58.477993 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:56:58.477955    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:00.057964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:00.057927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:01.741238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:01.741193    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:03.694345 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:03.694288    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:05.308290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:05.308225    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:06.958950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:06.958874    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:08.715900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:08.715862    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:10.408180 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:10.408096    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:12.440689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:12.440632    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:12.920093 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:12.920063    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:14.299704 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.299613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:14.563458 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.563381    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:14.632567 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.632533    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"]
Oct 30 17:57:14.637065 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.637040    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.660668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.660624    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"]
Oct 30 17:57:14.672671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.669897    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:57:14.672671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.669946    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:57:14.772236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.772195    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzn8\" (UniqueName: \"kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.772439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.772248    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.772439 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.772410    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.874342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.874311    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzn8\" (UniqueName: \"kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.874554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.874353    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.874554 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.874408    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.875150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.875127    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.876916 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.876893    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.897516 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.897490    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzn8\" (UniqueName: \"kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8\") pod \"maas-api-revoke-sub-bcb8f8e3-8zzr6\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:14.955731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:14.955709    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:15.245225 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:57:15.245151    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:57:15.267388 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:15.267361    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"]
Oct 30 17:57:15.268435 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:57:15.268409    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8feb148_6af9_4a31_a4e2_d90d1b2c0592.slice/crio-0fbd227b4947117c3e576b99ed85d6a75b7c27c19e1e609448f4a990d1fbfd2c WatchSource:0}: Error finding container 0fbd227b4947117c3e576b99ed85d6a75b7c27c19e1e609448f4a990d1fbfd2c: Status 404 returned error can't find the container with id 0fbd227b4947117c3e576b99ed85d6a75b7c27c19e1e609448f4a990d1fbfd2c
Oct 30 17:57:15.749271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:15.749228    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6" event={"ID":"f8feb148-6af9-4a31-a4e2-d90d1b2c0592","Type":"ContainerStarted","Data":"677a87b179687150a19e59fb157a812ec786753199469365ed7db441e1d20b5b"}
Oct 30 17:57:15.749720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:15.749321    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6" event={"ID":"f8feb148-6af9-4a31-a4e2-d90d1b2c0592","Type":"ContainerStarted","Data":"0fbd227b4947117c3e576b99ed85d6a75b7c27c19e1e609448f4a990d1fbfd2c"}
Oct 30 17:57:15.789230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:15.789178    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6" podStartSLOduration=1.789163507 podStartE2EDuration="1.789163507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:15.788823778 +0000 UTC m=+2641.550867591" watchObservedRunningTime="2026-10-30 17:57:15.789163507 +0000 UTC m=+2641.551207320"
Oct 30 17:57:15.955495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:15.955461    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:16.761186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:16.761146    5534 generic.go:356] "Generic (PLEG): container finished" podID="f8feb148-6af9-4a31-a4e2-d90d1b2c0592" containerID="677a87b179687150a19e59fb157a812ec786753199469365ed7db441e1d20b5b" exitCode=0
Oct 30 17:57:16.761629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:16.761201    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6" event={"ID":"f8feb148-6af9-4a31-a4e2-d90d1b2c0592","Type":"ContainerDied","Data":"677a87b179687150a19e59fb157a812ec786753199469365ed7db441e1d20b5b"}
Oct 30 17:57:17.716447 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:17.716412    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:17.956221 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:17.956192    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:18.123236 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.123201    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8\" (UniqueName: \"kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8\") pod \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") "
Oct 30 17:57:18.123453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.123263    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca\") pod \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") "
Oct 30 17:57:18.123453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.123359    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca\") pod \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\" (UID: \"f8feb148-6af9-4a31-a4e2-d90d1b2c0592\") "
Oct 30 17:57:18.123771 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.123740    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca" pod "f8feb148-6af9-4a31-a4e2-d90d1b2c0592" (UID: "f8feb148-6af9-4a31-a4e2-d90d1b2c0592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:57:18.124381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.124360    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:18.125543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.125521    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca" pod "f8feb148-6af9-4a31-a4e2-d90d1b2c0592" (UID: "f8feb148-6af9-4a31-a4e2-d90d1b2c0592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:57:18.125716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.125694    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8" pod "f8feb148-6af9-4a31-a4e2-d90d1b2c0592" (UID: "f8feb148-6af9-4a31-a4e2-d90d1b2c0592"). InnerVolumeSpecName "kube-api-access-vlzn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:57:18.225487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.225451    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzn8\" (UniqueName: \"kubernetes.io/projected/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-kube-api-access-vlzn8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:18.225487 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.225480    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8feb148-6af9-4a31-a4e2-d90d1b2c0592-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:18.787478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.787389    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"
Oct 30 17:57:18.787478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.787418    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6" event={"ID":"f8feb148-6af9-4a31-a4e2-d90d1b2c0592","Type":"ContainerDied","Data":"0fbd227b4947117c3e576b99ed85d6a75b7c27c19e1e609448f4a990d1fbfd2c"}
Oct 30 17:57:18.787478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:18.787463    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fbd227b4947117c3e576b99ed85d6a75b7c27c19e1e609448f4a990d1fbfd2c"
Oct 30 17:57:19.191654 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:19.191618    5534 scope.go:122] "RemoveContainer" containerID="2fd1a18da53c05a2f1dba17991c10babd48a26ada32090ee2684b8914052af2c"
Oct 30 17:57:19.203021 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:19.203002    5534 scope.go:122] "RemoveContainer" containerID="4813d7ca5aa9542967c683dd01df3590502e318f73e9f91482351e726a54d774"
Oct 30 17:57:19.217109 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:19.217084    5534 scope.go:122] "RemoveContainer" containerID="96008a69d82648e8fa2b259a16fd4b365ef89b652b3a372a74f649718f00471b"
Oct 30 17:57:19.227772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:19.227750    5534 scope.go:122] "RemoveContainer" containerID="3602b9a0c9bd1bbc5aad4da060321145dd51872be995cc435af050e2123f4716"
Oct 30 17:57:19.299505 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:19.299469    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:20.965578 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:20.965540    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:21.881124 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:21.881090    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:22.765536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:22.765489    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:22.882531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:22.882496    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:23.001856 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:23.001821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:24.627401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.627366    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"]
Oct 30 17:57:24.628370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.628351    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f8feb148-6af9-4a31-a4e2-d90d1b2c0592" containerName="revoke-keys"
Oct 30 17:57:24.628429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.628372    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8feb148-6af9-4a31-a4e2-d90d1b2c0592" containerName="revoke-keys"
Oct 30 17:57:24.628562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.628548    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8feb148-6af9-4a31-a4e2-d90d1b2c0592" containerName="revoke-keys"
Oct 30 17:57:24.633637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.633609    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.650479 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.650436    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-21870f-serving-cert\""
Oct 30 17:57:24.653222 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.653198    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"]
Oct 30 17:57:24.722368 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.722323    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:24.787670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.787622    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.787670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.787671    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.787965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.787793    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.787965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.787835    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gs6k\" (UniqueName: \"kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.839827 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.839791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:24.889694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.889568    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.889694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.889659    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gs6k\" (UniqueName: \"kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.889977 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.889947    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.890061 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.889982    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.890867 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.890837    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.910673 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.900284    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.927361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.927234    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gs6k\" (UniqueName: \"kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.930360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.930334    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.954971 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.954928    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:24.955297 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:24.955261    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:25.037702 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.037624    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"]
Oct 30 17:57:25.046053 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.045541    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.063679 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.063076    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"]
Oct 30 17:57:25.066866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.066716    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:57:25.068811 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.066999    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:57:25.194928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.194855    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.195135 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.195114    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.195199 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.195174    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxxj\" (UniqueName: \"kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.279265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.279231    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"]
Oct 30 17:57:25.280136 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:57:25.280110    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c7f2b7_9a2a_4b50_8646_12ed8f29945c.slice/crio-e75aeadfc70641c2fafe0ed92656ef4c6f7efe42a0a965746f55a89f90b33f63 WatchSource:0}: Error finding container e75aeadfc70641c2fafe0ed92656ef4c6f7efe42a0a965746f55a89f90b33f63: Status 404 returned error can't find the container with id e75aeadfc70641c2fafe0ed92656ef4c6f7efe42a0a965746f55a89f90b33f63
Oct 30 17:57:25.296882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.296815    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.296882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.296874    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxxj\" (UniqueName: \"kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.297057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.297016    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.297685 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.297625    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.299470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.299439    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.319607 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.319585    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxxj\" (UniqueName: \"kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj\") pod \"maas-api-revoke-sub-055514d9-7b2j4\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.379759 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.379728    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:25.708525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.708498    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"]
Oct 30 17:57:25.709170 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:57:25.709147    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b6c791_b042_4d23_8864_3f3637848091.slice/crio-570a0d86be889d8954138a3eba4dc5767a7387ad8ac20b80b3fafc68a913ed8d WatchSource:0}: Error finding container 570a0d86be889d8954138a3eba4dc5767a7387ad8ac20b80b3fafc68a913ed8d: Status 404 returned error can't find the container with id 570a0d86be889d8954138a3eba4dc5767a7387ad8ac20b80b3fafc68a913ed8d
Oct 30 17:57:25.935401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.935365    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4" event={"ID":"22b6c791-b042-4d23-8864-3f3637848091","Type":"ContainerStarted","Data":"570a0d86be889d8954138a3eba4dc5767a7387ad8ac20b80b3fafc68a913ed8d"}
Oct 30 17:57:25.937436 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.937406    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn" event={"ID":"46c7f2b7-9a2a-4b50-8646-12ed8f29945c","Type":"ContainerStarted","Data":"6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b"}
Oct 30 17:57:25.937584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.937461    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn" event={"ID":"46c7f2b7-9a2a-4b50-8646-12ed8f29945c","Type":"ContainerStarted","Data":"e75aeadfc70641c2fafe0ed92656ef4c6f7efe42a0a965746f55a89f90b33f63"}
Oct 30 17:57:25.937584 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.937545    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:25.983728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:25.983659    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn" podStartSLOduration=1.57662458 podStartE2EDuration="1.983622401s" totalImagesPullingTime="406.997821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:25.979548209 +0000 UTC m=+2651.741592023" watchObservedRunningTime="2026-10-30 17:57:25.983622401 +0000 UTC m=+2651.745666215"
Oct 30 17:57:26.533557 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:26.533528    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:26.947273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:26.947237    5534 generic.go:356] "Generic (PLEG): container finished" podID="22b6c791-b042-4d23-8864-3f3637848091" containerID="b6efbfc257877b85d8e08010e4b100aa0734bff26e012269e552d52c1c42d0cf" exitCode=0
Oct 30 17:57:26.947764 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:26.947321    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4" event={"ID":"22b6c791-b042-4d23-8864-3f3637848091","Type":"ContainerDied","Data":"b6efbfc257877b85d8e08010e4b100aa0734bff26e012269e552d52c1c42d0cf"}
Oct 30 17:57:28.123196 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.123153    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:28.132421 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.132395    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:28.252098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.252011    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca\") pod \"22b6c791-b042-4d23-8864-3f3637848091\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") "
Oct 30 17:57:28.252270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.252141    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj\" (UniqueName: \"kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj\") pod \"22b6c791-b042-4d23-8864-3f3637848091\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") "
Oct 30 17:57:28.252361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.252278    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca\") pod \"22b6c791-b042-4d23-8864-3f3637848091\" (UID: \"22b6c791-b042-4d23-8864-3f3637848091\") "
Oct 30 17:57:28.252748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.252684    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca" pod "22b6c791-b042-4d23-8864-3f3637848091" (UID: "22b6c791-b042-4d23-8864-3f3637848091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:57:28.253015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.252998    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b6c791-b042-4d23-8864-3f3637848091-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:28.254374 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.254337    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca" pod "22b6c791-b042-4d23-8864-3f3637848091" (UID: "22b6c791-b042-4d23-8864-3f3637848091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:57:28.254948 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.254921    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj" pod "22b6c791-b042-4d23-8864-3f3637848091" (UID: "22b6c791-b042-4d23-8864-3f3637848091"). InnerVolumeSpecName "kube-api-access-njxxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:57:28.354932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.354897    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b6c791-b042-4d23-8864-3f3637848091-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:28.354932 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.354933    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxxj\" (UniqueName: \"kubernetes.io/projected/22b6c791-b042-4d23-8864-3f3637848091-kube-api-access-njxxj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:28.965950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.965921    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"
Oct 30 17:57:28.965950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.965929    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4" event={"ID":"22b6c791-b042-4d23-8864-3f3637848091","Type":"ContainerDied","Data":"570a0d86be889d8954138a3eba4dc5767a7387ad8ac20b80b3fafc68a913ed8d"}
Oct 30 17:57:28.966168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:28.965983    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570a0d86be889d8954138a3eba4dc5767a7387ad8ac20b80b3fafc68a913ed8d"
Oct 30 17:57:29.961351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:29.961312    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:31.739965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:31.739935    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:31.956695 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:31.955937    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 17:57:33.269657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:33.269597    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:33.393086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:33.393051    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:35.002118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:35.002084    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:35.631240 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:35.631199    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:36.102984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:36.102953    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:36.711418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:36.711382    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:36.849859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:36.849824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:37.178576 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:37.178544    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:38.379337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:38.379303    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:38.731997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:38.731916    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:39.038119 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.038031    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:39.563668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.560752    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"]
Oct 30 17:57:39.563668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.561936    5534 cpu_manager.go:414] "containerMap: removing container" podUID="22b6c791-b042-4d23-8864-3f3637848091" containerName="revoke-keys"
Oct 30 17:57:39.563668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.561957    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b6c791-b042-4d23-8864-3f3637848091" containerName="revoke-keys"
Oct 30 17:57:39.563668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.562133    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b6c791-b042-4d23-8864-3f3637848091" containerName="revoke-keys"
Oct 30 17:57:39.567186 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.567163    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.581169 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.581138    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"]
Oct 30 17:57:39.611103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.611072    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:39.701670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.698380    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.728741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.728701    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.729134 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.729112    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.729394 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.729341    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.729523 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.729507    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzj5\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.729661 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.729628    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.780881 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.741887    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-455e4a-openshift-default-dockercfg-5hvtp\""
Oct 30 17:57:39.788800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.788767    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.821924 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.821802    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.822211 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.822168    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952290 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952249    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952348    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952399    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952520 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952488    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952516    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzj5\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952752 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952585    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952791    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952870 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952852    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.952984 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.952890    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.953878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.953467    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.953878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.953575    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.954371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.954303    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.954371 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.954327    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.954852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.954831    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.958610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.958584    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.959923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.959892    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.983407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.983366    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:39.983981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:39.983947    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzj5\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5\") pod \"e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") " pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:40.165664 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.163035    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:40.172667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.166886    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:40.880614 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.880576    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:40.994391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.994362    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"]
Oct 30 17:57:40.994532 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:57:40.994482    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe01f86_f520_4fb1_b7c3_9348cf833e9e.slice/crio-960bd1a5052f6f7350d97ae26687fff65b10997b0a3e2fd844c599153fbed91d WatchSource:0}: Error finding container 960bd1a5052f6f7350d97ae26687fff65b10997b0a3e2fd844c599153fbed91d: Status 404 returned error can't find the container with id 960bd1a5052f6f7350d97ae26687fff65b10997b0a3e2fd844c599153fbed91d
Oct 30 17:57:40.997317 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.997283    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:57:40.997410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.997349    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:57:40.997410 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:40.997376    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:57:41.051682 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:41.051608    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:41.309693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:41.303156    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:41.309693 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:41.305723    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" event={"ID":"cbe01f86-f520-4fb1-b7c3-9348cf833e9e","Type":"ContainerStarted","Data":"960bd1a5052f6f7350d97ae26687fff65b10997b0a3e2fd844c599153fbed91d"}
Oct 30 17:57:41.963136 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:41.963100    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"]
Oct 30 17:57:41.967843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:41.967818    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:41.982384 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:41.982357    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:57:42.002273 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.002169    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"]
Oct 30 17:57:42.063012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.062978    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdw4j\" (UniqueName: \"kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.063012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.063021    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.063299 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.063239    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.076976 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.076950    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:57:42.164438 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.164394    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.164632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.164612    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdw4j\" (UniqueName: \"kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.164740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.164676    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.165382 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.165312    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.167287 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.167260    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.187862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.187842    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdw4j\" (UniqueName: \"kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j\") pod \"maas-api-revoke-sub-d526a864-gvvpd\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.282276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.282186    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:42.317815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.317752    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" event={"ID":"cbe01f86-f520-4fb1-b7c3-9348cf833e9e","Type":"ContainerStarted","Data":"20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"}
Oct 30 17:57:42.367798 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.365857    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podStartSLOduration=3.365837576 podStartE2EDuration="3.365837576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:42.36372252 +0000 UTC m=+2668.125766334" watchObservedRunningTime="2026-10-30 17:57:42.365837576 +0000 UTC m=+2668.127881389"
Oct 30 17:57:42.573511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.570931    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:42.602034 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.601671    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"]
Oct 30 17:57:42.692464 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:42.692423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:43.168223 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.168185    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:43.174533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.174506    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:43.306824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.306792    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:43.327768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.327686    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd" event={"ID":"4b046dd1-e9c4-4362-8daa-818997c428b8","Type":"ContainerStarted","Data":"634bd815790de0cce0e436601221894f58135dc7244a69147b4ee246e15caf84"}
Oct 30 17:57:43.327768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.327741    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd" event={"ID":"4b046dd1-e9c4-4362-8daa-818997c428b8","Type":"ContainerStarted","Data":"6ec95c885b446da857c63f8c169c7e3086fea1ab3a88eb13dd484dfd34f31e9e"}
Oct 30 17:57:43.328042 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.328021    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:43.328990 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.328971    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:57:43.370524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:43.370470    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd" podStartSLOduration=2.370453331 podStartE2EDuration="2.370453331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:43.368162933 +0000 UTC m=+2669.130206748" watchObservedRunningTime="2026-10-30 17:57:43.370453331 +0000 UTC m=+2669.132497143"
Oct 30 17:57:44.337101 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:44.337070    5534 generic.go:356] "Generic (PLEG): container finished" podID="4b046dd1-e9c4-4362-8daa-818997c428b8" containerID="634bd815790de0cce0e436601221894f58135dc7244a69147b4ee246e15caf84" exitCode=0
Oct 30 17:57:44.337496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:44.337154    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd" event={"ID":"4b046dd1-e9c4-4362-8daa-818997c428b8","Type":"ContainerDied","Data":"634bd815790de0cce0e436601221894f58135dc7244a69147b4ee246e15caf84"}
Oct 30 17:57:44.798594 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:44.798557    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:44.912342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:44.912309    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:45.245951 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:57:45.245916    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:57:45.329259 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.329231    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:45.567515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.567488    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:45.645837 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.645803    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca\") pod \"4b046dd1-e9c4-4362-8daa-818997c428b8\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") "
Oct 30 17:57:45.646058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.645884    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca\") pod \"4b046dd1-e9c4-4362-8daa-818997c428b8\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") "
Oct 30 17:57:45.646141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.646069    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j\" (UniqueName: \"kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j\") pod \"4b046dd1-e9c4-4362-8daa-818997c428b8\" (UID: \"4b046dd1-e9c4-4362-8daa-818997c428b8\") "
Oct 30 17:57:45.646208 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.646153    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca" pod "4b046dd1-e9c4-4362-8daa-818997c428b8" (UID: "4b046dd1-e9c4-4362-8daa-818997c428b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:57:45.646820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.646788    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:45.648958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.648933    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca" pod "4b046dd1-e9c4-4362-8daa-818997c428b8" (UID: "4b046dd1-e9c4-4362-8daa-818997c428b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:57:45.651072 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.651040    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j" pod "4b046dd1-e9c4-4362-8daa-818997c428b8" (UID: "4b046dd1-e9c4-4362-8daa-818997c428b8"). InnerVolumeSpecName "kube-api-access-kdw4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:57:45.747934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.747898    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdw4j\" (UniqueName: \"kubernetes.io/projected/4b046dd1-e9c4-4362-8daa-818997c428b8-kube-api-access-kdw4j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:45.747934 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:45.747927    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b046dd1-e9c4-4362-8daa-818997c428b8-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:46.400965 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.400937    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"
Oct 30 17:57:46.401150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.400937    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd" event={"ID":"4b046dd1-e9c4-4362-8daa-818997c428b8","Type":"ContainerDied","Data":"6ec95c885b446da857c63f8c169c7e3086fea1ab3a88eb13dd484dfd34f31e9e"}
Oct 30 17:57:46.401150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.401076    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ec95c885b446da857c63f8c169c7e3086fea1ab3a88eb13dd484dfd34f31e9e"
Oct 30 17:57:46.638039 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.637945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:46.762697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.762661    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:46.807247 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.807215    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"]
Oct 30 17:57:46.808457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.808162    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4b046dd1-e9c4-4362-8daa-818997c428b8" containerName="revoke-keys"
Oct 30 17:57:46.808457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.808185    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b046dd1-e9c4-4362-8daa-818997c428b8" containerName="revoke-keys"
Oct 30 17:57:46.808841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.808427    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b046dd1-e9c4-4362-8daa-818997c428b8" containerName="revoke-keys"
Oct 30 17:57:46.813262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.813241    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:46.828892 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.828831    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-455e4a-dockercfg-wdbw9\""
Oct 30 17:57:46.831768 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.831744    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"]
Oct 30 17:57:46.857426 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:46.857396    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-455e4a\""
Oct 30 17:57:47.136175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.136142    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.136411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.136209    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frnd\" (UniqueName: \"kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.237383 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.237347    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.237563 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.237401    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frnd\" (UniqueName: \"kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.238123 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.238102    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.260982 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.260951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frnd\" (UniqueName: \"kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.345192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.345152    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:47.431152 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.431059    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:47.610686 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.610572    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:47.788991 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:57:47.788959    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd04965b_2490_4842_8afd_8586dd1131fa.slice/crio-fb9efb3fbad253e5c0c0a7fc9a69bdeabf41b9b397f56b6600a59008c028597f WatchSource:0}: Error finding container fb9efb3fbad253e5c0c0a7fc9a69bdeabf41b9b397f56b6600a59008c028597f: Status 404 returned error can't find the container with id fb9efb3fbad253e5c0c0a7fc9a69bdeabf41b9b397f56b6600a59008c028597f
Oct 30 17:57:47.789964 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:47.789938    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"]
Oct 30 17:57:48.436305 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:48.436271    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz" event={"ID":"fd04965b-2490-4842-8afd-8586dd1131fa","Type":"ContainerStarted","Data":"e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57"}
Oct 30 17:57:48.436531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:48.436325    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz" event={"ID":"fd04965b-2490-4842-8afd-8586dd1131fa","Type":"ContainerStarted","Data":"fb9efb3fbad253e5c0c0a7fc9a69bdeabf41b9b397f56b6600a59008c028597f"}
Oct 30 17:57:48.436531 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:48.436352    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:57:48.479816 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:48.479759    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz" podStartSLOduration=2.47974709 podStartE2EDuration="2.47974709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:48.475352499 +0000 UTC m=+2674.237396313" watchObservedRunningTime="2026-10-30 17:57:48.47974709 +0000 UTC m=+2674.241790902"
Oct 30 17:57:49.202512 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:49.202456    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:50.808762 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:50.808729    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:52.304327 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.304294    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"]
Oct 30 17:57:52.309070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.309048    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.337175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.337059    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:57:52.337175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.337101    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:57:52.343612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.343589    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"]
Oct 30 17:57:52.422729 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.422691    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77tt\" (UniqueName: \"kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.422888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.422736    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.423020 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.422994    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.429869 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.429821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:52.525387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.525357    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77tt\" (UniqueName: \"kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.525387 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.525393    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.525706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.525599    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.526364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.526331    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.528282 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.528263    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.548296 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.548274    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77tt\" (UniqueName: \"kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt\") pod \"maas-api-revoke-sub-00fc7e49-wzgsw\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.624466 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.624401    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:52.935920 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:57:52.935889    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8f063e_7038_4035_b14e_e2519fa2ec43.slice/crio-79bdb478638a3fff85b9afa8b3f6b05a433ec9fce25646ddcaea4df8fa5833ea WatchSource:0}: Error finding container 79bdb478638a3fff85b9afa8b3f6b05a433ec9fce25646ddcaea4df8fa5833ea: Status 404 returned error can't find the container with id 79bdb478638a3fff85b9afa8b3f6b05a433ec9fce25646ddcaea4df8fa5833ea
Oct 30 17:57:52.936073 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:52.935943    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"]
Oct 30 17:57:53.490177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:53.490139    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw" event={"ID":"8b8f063e-7038-4035-b14e-e2519fa2ec43","Type":"ContainerStarted","Data":"89774dfb7eda014bf944ed504d02a86962d8051cfd78f4a7a842fd24496d9fd8"}
Oct 30 17:57:53.490680 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:53.490224    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw" event={"ID":"8b8f063e-7038-4035-b14e-e2519fa2ec43","Type":"ContainerStarted","Data":"79bdb478638a3fff85b9afa8b3f6b05a433ec9fce25646ddcaea4df8fa5833ea"}
Oct 30 17:57:53.531041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:53.530950    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw" podStartSLOduration=1.53093637 podStartE2EDuration="1.53093637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:53.529831154 +0000 UTC m=+2679.291874970" watchObservedRunningTime="2026-10-30 17:57:53.53093637 +0000 UTC m=+2679.292980183"
Oct 30 17:57:54.033841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:54.033810    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"]
Oct 30 17:57:54.039692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:54.039667    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:54.045534 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:54.045514    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e19b616-ktjmv"]
Oct 30 17:57:54.376748 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:54.376713    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d267fc23-db31-4a8a-8cff-42cddb43a3f7" path="/var/lib/kubelet/pods/d267fc23-db31-4a8a-8cff-42cddb43a3f7/volumes"
Oct 30 17:57:54.503265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:54.503229    5534 generic.go:356] "Generic (PLEG): container finished" podID="8b8f063e-7038-4035-b14e-e2519fa2ec43" containerID="89774dfb7eda014bf944ed504d02a86962d8051cfd78f4a7a842fd24496d9fd8" exitCode=0
Oct 30 17:57:54.503739 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:54.503369    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw" event={"ID":"8b8f063e-7038-4035-b14e-e2519fa2ec43","Type":"ContainerDied","Data":"89774dfb7eda014bf944ed504d02a86962d8051cfd78f4a7a842fd24496d9fd8"}
Oct 30 17:57:55.701145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.701120    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:55.773540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.772410    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt\" (UniqueName: \"kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt\") pod \"8b8f063e-7038-4035-b14e-e2519fa2ec43\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") "
Oct 30 17:57:55.773540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.772607    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca\") pod \"8b8f063e-7038-4035-b14e-e2519fa2ec43\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") "
Oct 30 17:57:55.773540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.772676    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca\") pod \"8b8f063e-7038-4035-b14e-e2519fa2ec43\" (UID: \"8b8f063e-7038-4035-b14e-e2519fa2ec43\") "
Oct 30 17:57:55.773540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.773075    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca" pod "8b8f063e-7038-4035-b14e-e2519fa2ec43" (UID: "8b8f063e-7038-4035-b14e-e2519fa2ec43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:57:55.773540 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.773358    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:55.776476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.776294    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca" pod "8b8f063e-7038-4035-b14e-e2519fa2ec43" (UID: "8b8f063e-7038-4035-b14e-e2519fa2ec43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:57:55.778344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.778313    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt" pod "8b8f063e-7038-4035-b14e-e2519fa2ec43" (UID: "8b8f063e-7038-4035-b14e-e2519fa2ec43"). InnerVolumeSpecName "kube-api-access-b77tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:57:55.874961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.874929    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f063e-7038-4035-b14e-e2519fa2ec43-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:55.874961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.874962    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b77tt\" (UniqueName: \"kubernetes.io/projected/8b8f063e-7038-4035-b14e-e2519fa2ec43-kube-api-access-b77tt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:57:55.896499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:55.896461    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:56.521558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:56.521525    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"
Oct 30 17:57:56.521558 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:56.521536    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw" event={"ID":"8b8f063e-7038-4035-b14e-e2519fa2ec43","Type":"ContainerDied","Data":"79bdb478638a3fff85b9afa8b3f6b05a433ec9fce25646ddcaea4df8fa5833ea"}
Oct 30 17:57:56.522010 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:56.521582    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79bdb478638a3fff85b9afa8b3f6b05a433ec9fce25646ddcaea4df8fa5833ea"
Oct 30 17:57:57.609027 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:57.608988    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:59.290386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:59.290344    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:57:59.447200 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:57:59.447169    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:58:00.955573 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:00.955540    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:02.777937 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:02.777905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:04.373905 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:04.373874    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:05.008292 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:05.008252    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:05.588069 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:05.588035    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:06.056987 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:06.056954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:06.177667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:06.177436    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:06.309728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:06.307708    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:08.320161 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:08.320131    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:10.009813 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:10.009754    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:10.129999 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:10.129965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:10.983285 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:10.983222    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:11.111852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:11.111809    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:11.230568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:11.230535    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:12.069363 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:12.069323    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:12.187543 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:12.187507    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:12.311108 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:12.311066    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:14.002568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:14.002538    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:15.253273 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:15.253236    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:58:15.647979 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:15.647940    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:17.619006 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:17.618965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:19.030535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:19.030504    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"]
Oct 30 17:58:19.041747 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:19.041710    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14abcaa3-swz57"]
Oct 30 17:58:19.275727 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:19.275699    5534 scope.go:122] "RemoveContainer" containerID="bc61465656aac5fc6f176f5012b7849b75def8484f905896ec7c2cc2623fdf2d"
Oct 30 17:58:19.287205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:19.287106    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:19.287792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:19.287767    5534 scope.go:122] "RemoveContainer" containerID="445104d711b45bf74219f5728808bf81189d89bfc152e2cf2f81d6ca9e765ff2"
Oct 30 17:58:19.301409 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:19.301385    5534 scope.go:122] "RemoveContainer" containerID="374ba5d2f347e018b4ced05b6c97801dcaad9c234f463818b063e3669b96dab4"
Oct 30 17:58:20.376956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:20.376923    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107b59fd-c805-400b-b944-35d563b1020f" path="/var/lib/kubelet/pods/107b59fd-c805-400b-b944-35d563b1020f/volumes"
Oct 30 17:58:20.873508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:20.873460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:20.989972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:20.989939    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:22.006461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:22.006426    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:23.018276 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:23.018219    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:24.707801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:24.707702    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:25.059295 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:25.058701    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"]
Oct 30 17:58:25.087668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:25.086814    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d72bc1e-n6q2n"]
Oct 30 17:58:26.182517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:26.182475    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:26.420537 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:26.420485    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903d68bb-60cc-49f9-b93b-f452378a86dd" path="/var/lib/kubelet/pods/903d68bb-60cc-49f9-b93b-f452378a86dd/volumes"
Oct 30 17:58:26.427207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:26.427180    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:26.541914 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:26.541834    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:26.840812 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:26.840782    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:28.896669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:28.894387    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:30.584262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:30.584226    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:32.509848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:32.509818    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:33.391697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:33.391624    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:34.176629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:34.176574    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"]
Oct 30 17:58:34.205600 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:34.205566    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:34.338028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:34.337993    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:35.895288 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:35.895241    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:37.797939 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:37.797853    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:38.005429 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:38.005346    5534 generic.go:356] "Generic (PLEG): container finished" podID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerID="e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb" exitCode=0
Oct 30 17:58:38.005581 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:38.005420    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" event={"ID":"3612c751-1efc-4498-b1aa-2b8e47e45e8c","Type":"ContainerDied","Data":"e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"}
Oct 30 17:58:38.006024 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:38.006007    5534 scope.go:122] "RemoveContainer" containerID="e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"
Oct 30 17:58:38.008041 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:38.008016    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:58:38.008115 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:38.008084    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:58:38.008158 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:38.008115    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:58:39.016051 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:39.016013    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" event={"ID":"3612c751-1efc-4498-b1aa-2b8e47e45e8c","Type":"ContainerStarted","Data":"7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708"}
Oct 30 17:58:39.268148 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:39.268058    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:58:39.268508 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:39.268484    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 17:58:39.268731 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:39.268685    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 17:58:39.612546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:39.612516    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:40.275672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.270714    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 17:58:40.275672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.270774    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 17:58:40.719632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.710851    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8848-c6a580"]
Oct 30 17:58:40.719632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.712580    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8b8f063e-7038-4035-b14e-e2519fa2ec43" containerName="revoke-keys"
Oct 30 17:58:40.719632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.712603    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8f063e-7038-4035-b14e-e2519fa2ec43" containerName="revoke-keys"
Oct 30 17:58:40.719632 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.712982    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8f063e-7038-4035-b14e-e2519fa2ec43" containerName="revoke-keys"
Oct 30 17:58:40.721058 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.721036    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:40.731900 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.731874    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-c6a580"]
Oct 30 17:58:40.739688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.736703    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5c6gd\""
Oct 30 17:58:40.841435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.841375    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxsbs\" (UniqueName: \"kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs\") pod \"test-curl-8848-c6a580\" (UID: \"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38\") " pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:40.943753 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.943715    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsbs\" (UniqueName: \"kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs\") pod \"test-curl-8848-c6a580\" (UID: \"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38\") " pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:40.990992 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:40.990819    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsbs\" (UniqueName: \"kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs\") pod \"test-curl-8848-c6a580\" (UID: \"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38\") " pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:41.038635 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:41.038586    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:41.284717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:41.281394    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 17:58:41.284717 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:41.281458    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 17:58:41.384676 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:41.384184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:41.405698 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:41.404329    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-c6a580"]
Oct 30 17:58:42.042476 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:42.042446    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-c6a580" event={"ID":"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38","Type":"ContainerStarted","Data":"eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94"}
Oct 30 17:58:42.042726 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:42.042711    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-c6a580" event={"ID":"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38","Type":"ContainerStarted","Data":"ea10cd477a30662f04fba20e5f8756e0c483eab721dda64a0be08cbf143200c9"}
Oct 30 17:58:42.089862 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:42.089806    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8848-c6a580" podStartSLOduration=2.089791197 podStartE2EDuration="2.089791197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:42.084927753 +0000 UTC m=+2727.846971567" watchObservedRunningTime="2026-10-30 17:58:42.089791197 +0000 UTC m=+2727.851835009"
Oct 30 17:58:42.268066 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:42.268032    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 17:58:42.268359 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:42.268333    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 17:58:42.931670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:42.931613    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:43.003470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.003438    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8848-c6a580"]
Oct 30 17:58:43.005564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.005538    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8848-c6a580"]
Oct 30 17:58:43.189720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.179139    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8848-c06739"]
Oct 30 17:58:43.189720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.180283    5534 cpu_manager.go:414] "containerMap: removing container" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" containerName="test-curl-8848-c6a580"
Oct 30 17:58:43.189720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.180303    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" containerName="test-curl-8848-c6a580"
Oct 30 17:58:43.189720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.180468    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" containerName="test-curl-8848-c6a580"
Oct 30 17:58:43.209855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.209820    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-c06739"]
Oct 30 17:58:43.210035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.209936    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:58:43.284696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.274058    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 17:58:43.284696 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.274120    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 17:58:43.322950 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.322912    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5z2f\" (UniqueName: \"kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f\") pod \"test-curl-8848-c06739\" (UID: \"e767c53d-034d-4010-9e95-554aba493871\") " pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:58:43.424972 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.424935    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5z2f\" (UniqueName: \"kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f\") pod \"test-curl-8848-c06739\" (UID: \"e767c53d-034d-4010-9e95-554aba493871\") " pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:58:43.442150 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.442066    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:43.448767 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.448734    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5z2f\" (UniqueName: \"kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f\") pod \"test-curl-8848-c06739\" (UID: \"e767c53d-034d-4010-9e95-554aba493871\") " pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:58:43.528407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.528377    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:58:43.884841 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:43.884813    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-c06739"]
Oct 30 17:58:44.090418 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:44.090376    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-c06739" event={"ID":"e767c53d-034d-4010-9e95-554aba493871","Type":"ContainerStarted","Data":"5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5"}
Oct 30 17:58:44.091008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:44.090432    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-c06739" event={"ID":"e767c53d-034d-4010-9e95-554aba493871","Type":"ContainerStarted","Data":"a5ba5a2e5e5e4dafb570beb8a64778641550d9cee18418714c81cab235b14adb"}
Oct 30 17:58:44.091008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:44.090517    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8848-c6a580" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" containerName="test-curl-8848-c6a580" containerID="cri-o://eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94" gracePeriod=2
Oct 30 17:58:44.104412 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:44.104384    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-c6a580\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:44.142743 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:44.142660    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8848-c06739" podStartSLOduration=1.142631934 podStartE2EDuration="1.142631934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:44.141190178 +0000 UTC m=+2729.903233992" watchObservedRunningTime="2026-10-30 17:58:44.142631934 +0000 UTC m=+2729.904675744"
Oct 30 17:58:44.168528 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:44.168489    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-c6a580\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:44.263855 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:44.263818    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 17:58:44.264078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:44.263885    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 17:58:44.384087 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:44.384049    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-c6a580\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:45.163831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:45.163799    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:45.262616 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:45.262576    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:58:45.268772 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:45.268748    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:58:45.269209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:45.269187    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:58:45.269848 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:45.269830    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 17:58:46.308843 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:46.308816    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:46.370577 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:46.370491    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs\" (UniqueName: \"kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs\") pod \"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38\" (UID: \"243d5cc0-39ab-4153-a4b1-cbc49c7ffe38\") "
Oct 30 17:58:46.372883 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:46.372858    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs" pod "243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" (UID: "243d5cc0-39ab-4153-a4b1-cbc49c7ffe38"). InnerVolumeSpecName "kube-api-access-kxsbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:58:46.472968 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:46.472934    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxsbs\" (UniqueName: \"kubernetes.io/projected/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38-kube-api-access-kxsbs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:46.800917 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:46.800825    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:47.118435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:47.118398    5534 generic.go:356] "Generic (PLEG): container finished" podID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" containerID="eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94" exitCode=137
Oct 30 17:58:47.118687 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:47.118475    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c6a580"
Oct 30 17:58:47.118763 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:47.118731    5534 scope.go:122] "RemoveContainer" containerID="eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94"
Oct 30 17:58:47.130546 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:47.130528    5534 scope.go:122] "RemoveContainer" containerID="eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94"
Oct 30 17:58:47.130862 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:47.130843    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94\": container with ID starting with eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94 not found: ID does not exist" containerID="eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94"
Oct 30 17:58:47.130913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:47.130870    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94"} err="failed to get container status \"eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94\": rpc error: code = NotFound desc = could not find container \"eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94\": container with ID starting with eb708fbdd09297f3fee6ced624ad668cea9c011c89df06d18ad419f355aceb94 not found: ID does not exist"
Oct 30 17:58:47.571891 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:47.571810    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:48.372362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:48.372322    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243d5cc0-39ab-4153-a4b1-cbc49c7ffe38" path="/var/lib/kubelet/pods/243d5cc0-39ab-4153-a4b1-cbc49c7ffe38/volumes"
Oct 30 17:58:48.761078 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:48.760988    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:49.201797 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.201750    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57" podUID="4690a357-7337-4083-814c-690361db774e" containerName="main" containerID="cri-o://08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27" gracePeriod=45
Oct 30 17:58:49.499736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.499710    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:58:49.635580 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.635535    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache\") pod \"4690a357-7337-4083-814c-690361db774e\" (UID: \"4690a357-7337-4083-814c-690361db774e\") "
Oct 30 17:58:49.635777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.635745    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs\") pod \"4690a357-7337-4083-814c-690361db774e\" (UID: \"4690a357-7337-4083-814c-690361db774e\") "
Oct 30 17:58:49.635838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.635816    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache" pod "4690a357-7337-4083-814c-690361db774e" (UID: "4690a357-7337-4083-814c-690361db774e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:58:49.635887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.635829    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home\") pod \"4690a357-7337-4083-814c-690361db774e\" (UID: \"4690a357-7337-4083-814c-690361db774e\") "
Oct 30 17:58:49.636071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636050    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home" pod "4690a357-7337-4083-814c-690361db774e" (UID: "4690a357-7337-4083-814c-690361db774e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:58:49.636153 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636052    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm\") pod \"4690a357-7337-4083-814c-690361db774e\" (UID: \"4690a357-7337-4083-814c-690361db774e\") "
Oct 30 17:58:49.636212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636158    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir\") pod \"4690a357-7337-4083-814c-690361db774e\" (UID: \"4690a357-7337-4083-814c-690361db774e\") "
Oct 30 17:58:49.636212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636198    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc\" (UniqueName: \"kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc\") pod \"4690a357-7337-4083-814c-690361db774e\" (UID: \"4690a357-7337-4083-814c-690361db774e\") "
Oct 30 17:58:49.636800 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636375    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir" pod "4690a357-7337-4083-814c-690361db774e" (UID: "4690a357-7337-4083-814c-690361db774e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:58:49.636959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636938    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:49.637031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636963    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:49.637031 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.636976    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:49.638478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.638456    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm" pod "4690a357-7337-4083-814c-690361db774e" (UID: "4690a357-7337-4083-814c-690361db774e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:58:49.638613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.638591    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs" pod "4690a357-7337-4083-814c-690361db774e" (UID: "4690a357-7337-4083-814c-690361db774e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:58:49.639230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.639209    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc" pod "4690a357-7337-4083-814c-690361db774e" (UID: "4690a357-7337-4083-814c-690361db774e"). InnerVolumeSpecName "kube-api-access-twxkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:58:49.738801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.738707    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4690a357-7337-4083-814c-690361db774e-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:49.738801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.738739    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4690a357-7337-4083-814c-690361db774e-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:49.738801 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.738748    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxkc\" (UniqueName: \"kubernetes.io/projected/4690a357-7337-4083-814c-690361db774e-kube-api-access-twxkc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:58:49.969352 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:49.969312    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:50.134667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.134528    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:50.159666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.158898    5534 generic.go:356] "Generic (PLEG): container finished" podID="4690a357-7337-4083-814c-690361db774e" containerID="08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27" exitCode=0
Oct 30 17:58:50.159666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.158969    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57" event={"ID":"4690a357-7337-4083-814c-690361db774e","Type":"ContainerDied","Data":"08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27"}
Oct 30 17:58:50.159666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.159009    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57" event={"ID":"4690a357-7337-4083-814c-690361db774e","Type":"ContainerDied","Data":"b54c29504125d70e90148af216b6260e6dc9b481a3ad20f2fb6caee4894c3779"}
Oct 30 17:58:50.159666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.159047    5534 scope.go:122] "RemoveContainer" containerID="08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27"
Oct 30 17:58:50.159666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.159266    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"
Oct 30 17:58:50.186362 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.186298    5534 scope.go:122] "RemoveContainer" containerID="08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27"
Oct 30 17:58:50.186798 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:50.186764    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27\": container with ID starting with 08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27 not found: ID does not exist" containerID="08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27"
Oct 30 17:58:50.186912 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.186810    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27"} err="failed to get container status \"08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27\": rpc error: code = NotFound desc = could not find container \"08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27\": container with ID starting with 08bf7f4f0301d8047ffc568c17b038b94db574e7fc67f6eba33aac3075e97e27 not found: ID does not exist"
Oct 30 17:58:50.215997 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.215971    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"]
Oct 30 17:58:50.219396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.219370    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/auth-test-model-a2a6cb-kserve-7c785b57d9-2tj57"]
Oct 30 17:58:50.434986 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.434905    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4690a357-7337-4083-814c-690361db774e" path="/var/lib/kubelet/pods/4690a357-7337-4083-814c-690361db774e/volumes"
Oct 30 17:58:50.548671 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.548062    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"]
Oct 30 17:58:50.548876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.548804    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4690a357-7337-4083-814c-690361db774e" containerName="main"
Oct 30 17:58:50.548876 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.548817    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4690a357-7337-4083-814c-690361db774e" containerName="main"
Oct 30 17:58:50.549209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.549034    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4690a357-7337-4083-814c-690361db774e" containerName="main"
Oct 30 17:58:50.555396 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.555368    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.572333 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.572306    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"]
Oct 30 17:58:50.627981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.627948    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-21870f\""
Oct 30 17:58:50.628170 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.628040    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-21870f-dockercfg-qvphd\""
Oct 30 17:58:50.663264 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.663228    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ms5p\" (UniqueName: \"kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p\") pod \"payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.663553 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.663529    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.765443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.765359    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ms5p\" (UniqueName: \"kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p\") pod \"payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.765443 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.765417    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.766220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.766198    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.788923 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.788891    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ms5p\" (UniqueName: \"kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p\") pod \"payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:50.872331 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:50.872293    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:51.056958 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:51.056827    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:51.184882 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:51.184855    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"]
Oct 30 17:58:52.036610 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:52.036569    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"]
Oct 30 17:58:52.050533 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:52.050492    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e6a9bc-xnrb2"]
Oct 30 17:58:52.192694 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:52.184005    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" event={"ID":"a017e36e-4f65-4737-a6e0-965ef6370a1d","Type":"ContainerStarted","Data":"62934a7761ae5d92c8ce07350b883a93efe7f83c06a5ae623b9462f67b92b8a5"}
Oct 30 17:58:52.221598 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:52.221561    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:52.377408 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:52.377348    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1f8e56-5273-4792-af5c-38c27c304ccd" path="/var/lib/kubelet/pods/8d1f8e56-5273-4792-af5c-38c27c304ccd/volumes"
Oct 30 17:58:52.802190 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:52.802102    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:53.193709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:53.193630    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" event={"ID":"a017e36e-4f65-4737-a6e0-965ef6370a1d","Type":"ContainerStarted","Data":"dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"}
Oct 30 17:58:53.194184 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:53.193870    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:58:53.237572 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:53.237508    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" podStartSLOduration=1.5139483459999998 podStartE2EDuration="3.237492097s" totalImagesPullingTime="1.723543751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:53.234072288 +0000 UTC m=+2738.996116105" watchObservedRunningTime="2026-10-30 17:58:53.237492097 +0000 UTC m=+2738.999535910"
Oct 30 17:58:54.530386 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:54.530356    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:54.640889 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:54.640846    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:54.756030 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:54.755990    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:56.286830 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:56.286797    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:57.943908 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:57.943877    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:59.317350 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:59.317317    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8848-c06739"]
Oct 30 17:58:59.317909 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:59.317535    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8848-c06739" podUID="e767c53d-034d-4010-9e95-554aba493871" containerName="test-curl-8848-c06739" containerID="cri-o://5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5" gracePeriod=2
Oct 30 17:58:59.320980 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:59.320955    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8848-c06739"]
Oct 30 17:58:59.330930 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:58:59.330893    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-c06739\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="e767c53d-034d-4010-9e95-554aba493871" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:58:59.661565 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:59.661517    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:58:59.835770 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:58:59.835736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:00.003701 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.003611    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"]
Oct 30 17:59:00.004361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.004346    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e767c53d-034d-4010-9e95-554aba493871" containerName="test-curl-8848-c06739"
Oct 30 17:59:00.004406 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.004362    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e767c53d-034d-4010-9e95-554aba493871" containerName="test-curl-8848-c06739"
Oct 30 17:59:00.004478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.004469    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e767c53d-034d-4010-9e95-554aba493871" containerName="test-curl-8848-c06739"
Oct 30 17:59:00.011224 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.011197    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.016688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.016630    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.016792 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.016758    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvn7k\" (UniqueName: \"kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.016887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.016872    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.024542 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:59:00.024517    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-c06739\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="e767c53d-034d-4010-9e95-554aba493871" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:59:00.032165 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.032141    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:00.032376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.032164    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"]
Oct 30 17:59:00.036703 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.036681    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:59:00.036823 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.036777    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:59:00.117893 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.117858    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.118118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.117922    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvn7k\" (UniqueName: \"kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.118118 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.117985    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.118689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.118667    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.120494 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.120471    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.141035 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.141009    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvn7k\" (UniqueName: \"kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k\") pod \"maas-api-revoke-sub-c4a74ffa-5kdgk\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.326906 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.326820    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:00.647194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:00.647158    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"]
Oct 30 17:59:00.648290 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:00.648257    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ee91de_cf0e_4a76_94c0_d222b5549189.slice/crio-11ac86fda33acc3175e75b91d7f36b1737a0083b37372b897e34753c4f56fddf WatchSource:0}: Error finding container 11ac86fda33acc3175e75b91d7f36b1737a0083b37372b897e34753c4f56fddf: Status 404 returned error can't find the container with id 11ac86fda33acc3175e75b91d7f36b1737a0083b37372b897e34753c4f56fddf
Oct 30 17:59:01.263244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.263140    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk" event={"ID":"c1ee91de-cf0e-4a76-94c0-d222b5549189","Type":"ContainerStarted","Data":"8dd26c22094dca982cbd2854e2d8d8524e01253caae236bb853e13915e6c801c"}
Oct 30 17:59:01.263244 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.263241    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk" event={"ID":"c1ee91de-cf0e-4a76-94c0-d222b5549189","Type":"ContainerStarted","Data":"11ac86fda33acc3175e75b91d7f36b1737a0083b37372b897e34753c4f56fddf"}
Oct 30 17:59:01.304470 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.304420    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk" podStartSLOduration=2.304406746 podStartE2EDuration="2.304406746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:01.30255356 +0000 UTC m=+2747.064597388" watchObservedRunningTime="2026-10-30 17:59:01.304406746 +0000 UTC m=+2747.066450558"
Oct 30 17:59:01.342805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.342762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:01.579114 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.579090    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:59:01.638046 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.638017    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f\" (UniqueName: \"kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f\") pod \"e767c53d-034d-4010-9e95-554aba493871\" (UID: \"e767c53d-034d-4010-9e95-554aba493871\") "
Oct 30 17:59:01.640398 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.640369    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f" pod "e767c53d-034d-4010-9e95-554aba493871" (UID: "e767c53d-034d-4010-9e95-554aba493871"). InnerVolumeSpecName "kube-api-access-c5z2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:01.740160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:01.740103    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5z2f\" (UniqueName: \"kubernetes.io/projected/e767c53d-034d-4010-9e95-554aba493871-kube-api-access-c5z2f\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:02.276692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.276619    5534 generic.go:356] "Generic (PLEG): container finished" podID="e767c53d-034d-4010-9e95-554aba493871" containerID="5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5" exitCode=137
Oct 30 17:59:02.276910 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.276895    5534 scope.go:122] "RemoveContainer" containerID="5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5"
Oct 30 17:59:02.277057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.277043    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:59:02.289969 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.289932    5534 generic.go:356] "Generic (PLEG): container finished" podID="c1ee91de-cf0e-4a76-94c0-d222b5549189" containerID="8dd26c22094dca982cbd2854e2d8d8524e01253caae236bb853e13915e6c801c" exitCode=0
Oct 30 17:59:02.290550 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.290526    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk" event={"ID":"c1ee91de-cf0e-4a76-94c0-d222b5549189","Type":"ContainerDied","Data":"8dd26c22094dca982cbd2854e2d8d8524e01253caae236bb853e13915e6c801c"}
Oct 30 17:59:02.299793 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.299770    5534 scope.go:122] "RemoveContainer" containerID="5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5"
Oct 30 17:59:02.300073 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:59:02.300049    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5\": container with ID starting with 5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5 not found: ID does not exist" containerID="5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5"
Oct 30 17:59:02.300141 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.300086    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5"} err="failed to get container status \"5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5\": rpc error: code = NotFound desc = could not find container \"5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5\": container with ID starting with 5664ccb37ef401681c9b290069534229bf2e39b6ebadd0780b891f0c3b7616a5 not found: ID does not exist"
Oct 30 17:59:02.346593 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:59:02.345377    5534 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-c06739\" is forbidden: User \"system:node:ip-10-0-135-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-170.ec2.internal' and this object" podUID="e767c53d-034d-4010-9e95-554aba493871" pod="opendatahub/test-curl-8848-c06739"
Oct 30 17:59:02.372884 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:02.372845    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e767c53d-034d-4010-9e95-554aba493871" path="/var/lib/kubelet/pods/e767c53d-034d-4010-9e95-554aba493871/volumes"
Oct 30 17:59:03.033815 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.033784    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:03.302530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.302433    5534 generic.go:356] "Generic (PLEG): container finished" podID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerID="dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09" exitCode=1
Oct 30 17:59:03.302530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.302507    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" event={"ID":"a017e36e-4f65-4737-a6e0-965ef6370a1d","Type":"ContainerDied","Data":"dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"}
Oct 30 17:59:03.303137 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.303112    5534 scope.go:122] "RemoveContainer" containerID="dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"
Oct 30 17:59:03.498927 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.498900    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:03.578344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.577481    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca\") pod \"c1ee91de-cf0e-4a76-94c0-d222b5549189\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") "
Oct 30 17:59:03.578344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.577950    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca\") pod \"c1ee91de-cf0e-4a76-94c0-d222b5549189\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") "
Oct 30 17:59:03.578344 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.578085    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k\" (UniqueName: \"kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k\") pod \"c1ee91de-cf0e-4a76-94c0-d222b5549189\" (UID: \"c1ee91de-cf0e-4a76-94c0-d222b5549189\") "
Oct 30 17:59:03.579430 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.579407    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca" pod "c1ee91de-cf0e-4a76-94c0-d222b5549189" (UID: "c1ee91de-cf0e-4a76-94c0-d222b5549189"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:03.581461 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.581434    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca" pod "c1ee91de-cf0e-4a76-94c0-d222b5549189" (UID: "c1ee91de-cf0e-4a76-94c0-d222b5549189"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:59:03.603048 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.603009    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k" pod "c1ee91de-cf0e-4a76-94c0-d222b5549189" (UID: "c1ee91de-cf0e-4a76-94c0-d222b5549189"). InnerVolumeSpecName "kube-api-access-jvn7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:03.680402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.680369    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:03.680402 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.680403    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee91de-cf0e-4a76-94c0-d222b5549189-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:03.680697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:03.680421    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvn7k\" (UniqueName: \"kubernetes.io/projected/c1ee91de-cf0e-4a76-94c0-d222b5549189-kube-api-access-jvn7k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:04.313185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:04.313155    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"
Oct 30 17:59:04.313185 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:04.313177    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk" event={"ID":"c1ee91de-cf0e-4a76-94c0-d222b5549189","Type":"ContainerDied","Data":"11ac86fda33acc3175e75b91d7f36b1737a0083b37372b897e34753c4f56fddf"}
Oct 30 17:59:04.313495 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:04.313229    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ac86fda33acc3175e75b91d7f36b1737a0083b37372b897e34753c4f56fddf"
Oct 30 17:59:04.316220 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:04.316192    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" event={"ID":"a017e36e-4f65-4737-a6e0-965ef6370a1d","Type":"ContainerStarted","Data":"3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172"}
Oct 30 17:59:04.316414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:04.316397    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:59:04.911507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:04.911470    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:06.506373 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:06.506343    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:08.216888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:08.216852    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:10.339511 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.339460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:10.524535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.524502    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"]
Oct 30 17:59:10.525389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.525341    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c1ee91de-cf0e-4a76-94c0-d222b5549189" containerName="revoke-keys"
Oct 30 17:59:10.525463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.525390    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ee91de-cf0e-4a76-94c0-d222b5549189" containerName="revoke-keys"
Oct 30 17:59:10.525518 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.525504    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ee91de-cf0e-4a76-94c0-d222b5549189" containerName="revoke-keys"
Oct 30 17:59:10.529391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.529372    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.549498 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.549472    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"]
Oct 30 17:59:10.556084 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.556060    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:59:10.556265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.556175    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:59:10.598920 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.598880    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrh8\" (UniqueName: \"kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.599164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.599028    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.599164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.599108    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.726668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.714169    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrh8\" (UniqueName: \"kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.726668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.714310    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.726668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.714385    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.743310 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.735734    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.755665 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.746427    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.777663 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.772273    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrh8\" (UniqueName: \"kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8\") pod \"maas-api-revoke-sub-5538cf0e-bk77q\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:10.865370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:10.865270    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:11.686457 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:11.676716    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"]
Oct 30 17:59:12.667779 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:12.667660    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:12.708151 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:12.708118    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q" event={"ID":"fd7e10d3-c9a9-48c9-9003-89ca3e003443","Type":"ContainerStarted","Data":"94ad815b00dc02dac698db31370be66173bb310d74832a1377ad732271b90314"}
Oct 30 17:59:12.708608 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:12.708180    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q" event={"ID":"fd7e10d3-c9a9-48c9-9003-89ca3e003443","Type":"ContainerStarted","Data":"b322b53540f55ed5057e44345700970251b1325d51953508587b15937755ff57"}
Oct 30 17:59:12.748732 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:12.748638    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q" podStartSLOduration=2.748621812 podStartE2EDuration="2.748621812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:12.748174205 +0000 UTC m=+2758.510218019" watchObservedRunningTime="2026-10-30 17:59:12.748621812 +0000 UTC m=+2758.510665624"
Oct 30 17:59:12.872089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:12.872034    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:13.718177 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:13.718143    5534 generic.go:356] "Generic (PLEG): container finished" podID="fd7e10d3-c9a9-48c9-9003-89ca3e003443" containerID="94ad815b00dc02dac698db31370be66173bb310d74832a1377ad732271b90314" exitCode=0
Oct 30 17:59:13.718612 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:13.718233    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q" event={"ID":"fd7e10d3-c9a9-48c9-9003-89ca3e003443","Type":"ContainerDied","Data":"94ad815b00dc02dac698db31370be66173bb310d74832a1377ad732271b90314"}
Oct 30 17:59:14.814859 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:14.814823    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:15.027442 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.027409    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:15.108450 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.108414    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8\" (UniqueName: \"kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8\") pod \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") "
Oct 30 17:59:15.108629 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.108603    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca\") pod \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") "
Oct 30 17:59:15.108736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.108665    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca\") pod \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\" (UID: \"fd7e10d3-c9a9-48c9-9003-89ca3e003443\") "
Oct 30 17:59:15.109736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.109704    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca" pod "fd7e10d3-c9a9-48c9-9003-89ca3e003443" (UID: "fd7e10d3-c9a9-48c9-9003-89ca3e003443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:15.111605 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.111580    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca" pod "fd7e10d3-c9a9-48c9-9003-89ca3e003443" (UID: "fd7e10d3-c9a9-48c9-9003-89ca3e003443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:59:15.111751 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.111727    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8" pod "fd7e10d3-c9a9-48c9-9003-89ca3e003443" (UID: "fd7e10d3-c9a9-48c9-9003-89ca3e003443"). InnerVolumeSpecName "kube-api-access-pmrh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:15.211628 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.211588    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrh8\" (UniqueName: \"kubernetes.io/projected/fd7e10d3-c9a9-48c9-9003-89ca3e003443-kube-api-access-pmrh8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:15.211828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.211632    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:15.211828 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.211672    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7e10d3-c9a9-48c9-9003-89ca3e003443-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:15.250322 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:59:15.250238    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:59:15.325761 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.325723    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 17:59:15.573481 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.573291    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:15.738275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.738244    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"
Oct 30 17:59:15.738275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.738267    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q" event={"ID":"fd7e10d3-c9a9-48c9-9003-89ca3e003443","Type":"ContainerDied","Data":"b322b53540f55ed5057e44345700970251b1325d51953508587b15937755ff57"}
Oct 30 17:59:15.738637 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:15.738316    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b322b53540f55ed5057e44345700970251b1325d51953508587b15937755ff57"
Oct 30 17:59:16.819582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:16.819549    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"]
Oct 30 17:59:16.825132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:16.825094    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:16.939933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:16.939899    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:17.055510 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:17.055457    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:17.204706 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:17.202835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:18.902005 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:18.901951    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:19.214716 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.214599    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:19.364455 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.364420    5534 scope.go:122] "RemoveContainer" containerID="8b0569f64dbde6566082f3908e4131dba69129d35d17307730d74837c4a3ad93"
Oct 30 17:59:19.379160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.379133    5534 scope.go:122] "RemoveContainer" containerID="f4e80ccde5b22160f077856787f6a7ba3cb7df1d4913288be6a84ecae8a5efce"
Oct 30 17:59:19.394782 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.394760    5534 scope.go:122] "RemoveContainer" containerID="9237d4bc3e3cfb952e9392666bd2f4d821c6cd2040fe12f0110d0563dbb6c1b7"
Oct 30 17:59:19.632334 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.631316    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:19.697844 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.697796    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"]
Oct 30 17:59:19.699089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.699058    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fd7e10d3-c9a9-48c9-9003-89ca3e003443" containerName="revoke-keys"
Oct 30 17:59:19.699089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.699088    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7e10d3-c9a9-48c9-9003-89ca3e003443" containerName="revoke-keys"
Oct 30 17:59:19.699376 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.699317    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7e10d3-c9a9-48c9-9003-89ca3e003443" containerName="revoke-keys"
Oct 30 17:59:19.757265 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.757184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:19.819182 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.819142    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"]
Oct 30 17:59:19.819389 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.819309    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.834205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.834174    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e927d7-openshift-default-dockercfg-96l5k\""
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.892631    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.892757    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.892792    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.892822    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn96v\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.892861    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.892923    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.893019    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.893107    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.908666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.893150    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:19.996666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.996300    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.996935    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.997140    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.997909    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.997959    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.998013    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn96v\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.998055    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.998151    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.998282    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.998410    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:19.999868    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.000480    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.000840    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.001143    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.001888    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.010715 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.004120    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.031390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.031323    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.032255 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.032227    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn96v\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v\") pod \"e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") " pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.202864 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.202832    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:20.205852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.205824    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:20.598819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.598793    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:20.637071 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.637034    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:20.637195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.637121    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:20.637195 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.637161    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:20.637548 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.637520    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"]
Oct 30 17:59:20.786535 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:20.786493    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" event={"ID":"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a","Type":"ContainerStarted","Data":"83fb750f4f66e315f20d6053d2103020ea6b2f523d06f580659c67994e664374"}
Oct 30 17:59:21.406342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.406307    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"]
Oct 30 17:59:21.424346 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.424302    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.434206 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.432079    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"]
Oct 30 17:59:21.449315 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.448944    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:59:21.465564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.464679    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:59:21.521657 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.521608    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.521846 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.521763    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fnx\" (UniqueName: \"kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.521913 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.521886    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.623820 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.623774    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.624028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.623880    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fnx\" (UniqueName: \"kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.624028 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.623959    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.624874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.624844    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.626499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.626474    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.648004 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.647971    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fnx\" (UniqueName: \"kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx\") pod \"maas-api-revoke-sub-04d916ec-6j8mf\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.792961 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.792879    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:21.796003 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.795967    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" event={"ID":"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a","Type":"ContainerStarted","Data":"c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"}
Oct 30 17:59:21.820311 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.820278    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz" podUID="fd04965b-2490-4842-8afd-8586dd1131fa" containerName="payload-pre-processing" containerID="cri-o://e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57" gracePeriod=25
Oct 30 17:59:21.846514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:21.846456    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podStartSLOduration=2.846431731 podStartE2EDuration="2.846431731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:21.84425511 +0000 UTC m=+2767.606298925" watchObservedRunningTime="2026-10-30 17:59:21.846431731 +0000 UTC m=+2767.608475556"
Oct 30 17:59:22.144513 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.144476    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"]
Oct 30 17:59:22.144786 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:22.144763    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6b390_7f85_4a10_b29f_aa2b21a28815.slice/crio-cc6eb3f94f45f379d0e191f4c5021b0f2c0603f25fba50da5bd6dfb5b866fa55 WatchSource:0}: Error finding container cc6eb3f94f45f379d0e191f4c5021b0f2c0603f25fba50da5bd6dfb5b866fa55: Status 404 returned error can't find the container with id cc6eb3f94f45f379d0e191f4c5021b0f2c0603f25fba50da5bd6dfb5b866fa55
Oct 30 17:59:22.208602 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.206875    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:22.218075 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.217227    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:22.271597 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.271571    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:59:22.339944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.339900    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume\") pod \"fd04965b-2490-4842-8afd-8586dd1131fa\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") "
Oct 30 17:59:22.340164 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.340099    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd\" (UniqueName: \"kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd\") pod \"fd04965b-2490-4842-8afd-8586dd1131fa\" (UID: \"fd04965b-2490-4842-8afd-8586dd1131fa\") "
Oct 30 17:59:22.340351 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.340324    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume" pod "fd04965b-2490-4842-8afd-8586dd1131fa" (UID: "fd04965b-2490-4842-8afd-8586dd1131fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:22.341562 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.341534    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd04965b-2490-4842-8afd-8586dd1131fa-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:22.342907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.342875    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd" pod "fd04965b-2490-4842-8afd-8586dd1131fa" (UID: "fd04965b-2490-4842-8afd-8586dd1131fa"). InnerVolumeSpecName "kube-api-access-6frnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:22.428049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.428007    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:22.454145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.443271    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6frnd\" (UniqueName: \"kubernetes.io/projected/fd04965b-2490-4842-8afd-8586dd1131fa-kube-api-access-6frnd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:22.806365 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.806319    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf" event={"ID":"09b6b390-7f85-4a10-b29f-aa2b21a28815","Type":"ContainerStarted","Data":"7b6d8c4f2806804df14123bddbfbe7bb29172de7fbb9efcef0e275ad9b827327"}
Oct 30 17:59:22.806586 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.806399    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf" event={"ID":"09b6b390-7f85-4a10-b29f-aa2b21a28815","Type":"ContainerStarted","Data":"cc6eb3f94f45f379d0e191f4c5021b0f2c0603f25fba50da5bd6dfb5b866fa55"}
Oct 30 17:59:22.808043 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.808017    5534 generic.go:356] "Generic (PLEG): container finished" podID="fd04965b-2490-4842-8afd-8586dd1131fa" containerID="e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57" exitCode=0
Oct 30 17:59:22.808187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.808050    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"
Oct 30 17:59:22.808187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.808102    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz" event={"ID":"fd04965b-2490-4842-8afd-8586dd1131fa","Type":"ContainerDied","Data":"e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57"}
Oct 30 17:59:22.808187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.808162    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz" event={"ID":"fd04965b-2490-4842-8afd-8586dd1131fa","Type":"ContainerDied","Data":"fb9efb3fbad253e5c0c0a7fc9a69bdeabf41b9b397f56b6600a59008c028597f"}
Oct 30 17:59:22.808364 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.808192    5534 scope.go:122] "RemoveContainer" containerID="e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57"
Oct 30 17:59:22.808524 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.808501    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:22.810187 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.810165    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 17:59:22.821842 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.821607    5534 scope.go:122] "RemoveContainer" containerID="e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57"
Oct 30 17:59:22.822028 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:59:22.822005    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57\": container with ID starting with e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57 not found: ID does not exist" containerID="e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57"
Oct 30 17:59:22.822077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.822038    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57"} err="failed to get container status \"e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57\": rpc error: code = NotFound desc = could not find container \"e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57\": container with ID starting with e8ce55c5597ddd71b243faf16341c3ff6922c5028433c5ea66619f57dad47a57 not found: ID does not exist"
Oct 30 17:59:22.846777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.846732    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf" podStartSLOduration=1.846718617 podStartE2EDuration="1.846718617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:22.845809648 +0000 UTC m=+2768.607853462" watchObservedRunningTime="2026-10-30 17:59:22.846718617 +0000 UTC m=+2768.608762429"
Oct 30 17:59:22.887057 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.887017    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"]
Oct 30 17:59:22.892777 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:22.892313    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5c4c98cbcf-v4zjz"]
Oct 30 17:59:23.817086 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.817052    5534 generic.go:356] "Generic (PLEG): container finished" podID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerID="20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8" exitCode=0
Oct 30 17:59:23.817517 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.817125    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" event={"ID":"cbe01f86-f520-4fb1-b7c3-9348cf833e9e","Type":"ContainerDied","Data":"20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"}
Oct 30 17:59:23.817691 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.817673    5534 scope.go:122] "RemoveContainer" containerID="20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"
Oct 30 17:59:23.818967 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.818943    5534 generic.go:356] "Generic (PLEG): container finished" podID="09b6b390-7f85-4a10-b29f-aa2b21a28815" containerID="7b6d8c4f2806804df14123bddbfbe7bb29172de7fbb9efcef0e275ad9b827327" exitCode=0
Oct 30 17:59:23.819107 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.819035    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf" event={"ID":"09b6b390-7f85-4a10-b29f-aa2b21a28815","Type":"ContainerDied","Data":"7b6d8c4f2806804df14123bddbfbe7bb29172de7fbb9efcef0e275ad9b827327"}
Oct 30 17:59:23.819678 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.819636    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:23.819790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.819731    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:23.819790 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:23.819767    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:24.266757 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:24.266723    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:24.385515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:24.385477    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd04965b-2490-4842-8afd-8586dd1131fa" path="/var/lib/kubelet/pods/fd04965b-2490-4842-8afd-8586dd1131fa/volumes"
Oct 30 17:59:24.393090 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:24.393063    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:24.531633 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:24.531485    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:24.835278 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:24.833730    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" event={"ID":"cbe01f86-f520-4fb1-b7c3-9348cf833e9e","Type":"ContainerStarted","Data":"d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8"}
Oct 30 17:59:25.138973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.138945    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:25.180666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.180070    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:59:25.180666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.180210    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 30 17:59:25.180666 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.180257    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 30 17:59:25.209218 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.209177    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca\") pod \"09b6b390-7f85-4a10-b29f-aa2b21a28815\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") "
Oct 30 17:59:25.209403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.209315    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca\") pod \"09b6b390-7f85-4a10-b29f-aa2b21a28815\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") "
Oct 30 17:59:25.209403 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.209370    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx\" (UniqueName: \"kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx\") pod \"09b6b390-7f85-4a10-b29f-aa2b21a28815\" (UID: \"09b6b390-7f85-4a10-b29f-aa2b21a28815\") "
Oct 30 17:59:25.211019 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.210981    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca" pod "09b6b390-7f85-4a10-b29f-aa2b21a28815" (UID: "09b6b390-7f85-4a10-b29f-aa2b21a28815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:25.213077 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.213053    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx" pod "09b6b390-7f85-4a10-b29f-aa2b21a28815" (UID: "09b6b390-7f85-4a10-b29f-aa2b21a28815"). InnerVolumeSpecName "kube-api-access-45fnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:25.216568 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.216528    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca" pod "09b6b390-7f85-4a10-b29f-aa2b21a28815" (UID: "09b6b390-7f85-4a10-b29f-aa2b21a28815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:59:25.312262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.312229    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:25.312262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.312257    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6b390-7f85-4a10-b29f-aa2b21a28815-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:25.312262 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.312267    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45fnx\" (UniqueName: \"kubernetes.io/projected/09b6b390-7f85-4a10-b29f-aa2b21a28815-kube-api-access-45fnx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:25.844338 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.844304    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"
Oct 30 17:59:25.844803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.844335    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf" event={"ID":"09b6b390-7f85-4a10-b29f-aa2b21a28815","Type":"ContainerDied","Data":"cc6eb3f94f45f379d0e191f4c5021b0f2c0603f25fba50da5bd6dfb5b866fa55"}
Oct 30 17:59:25.844803 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:25.844395    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6eb3f94f45f379d0e191f4c5021b0f2c0603f25fba50da5bd6dfb5b866fa55"
Oct 30 17:59:26.002407 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.002374    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:26.124239 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.124146    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:26.191728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.179112    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 30 17:59:26.191728 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.191186    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 30 17:59:26.196810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.194133    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"]
Oct 30 17:59:26.196810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.196783    5534 cpu_manager.go:414] "containerMap: removing container" podUID="09b6b390-7f85-4a10-b29f-aa2b21a28815" containerName="revoke-keys"
Oct 30 17:59:26.196810 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.196803    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b6b390-7f85-4a10-b29f-aa2b21a28815" containerName="revoke-keys"
Oct 30 17:59:26.197168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.196831    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fd04965b-2490-4842-8afd-8586dd1131fa" containerName="payload-pre-processing"
Oct 30 17:59:26.197168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.196839    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd04965b-2490-4842-8afd-8586dd1131fa" containerName="payload-pre-processing"
Oct 30 17:59:26.197168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.197005    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b6b390-7f85-4a10-b29f-aa2b21a28815" containerName="revoke-keys"
Oct 30 17:59:26.197168 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.197024    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd04965b-2490-4842-8afd-8586dd1131fa" containerName="payload-pre-processing"
Oct 30 17:59:26.203067 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.203047    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"]
Oct 30 17:59:26.203358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.203305    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.218360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.218324    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e927d7-dockercfg-t7cgf\""
Oct 30 17:59:26.218360 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.218359    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e927d7\""
Oct 30 17:59:26.230188 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.230157    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.230496 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.230481    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4vc\" (UniqueName: \"kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.371668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.370562    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.371668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.370753    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4vc\" (UniqueName: \"kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.382722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.372104    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.545258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.539691    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4vc\" (UniqueName: \"kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.545258 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.542482    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"]
Oct 30 17:59:26.549435 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.549407    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"]
Oct 30 17:59:26.549604 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.549532    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.609541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.592103    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9fn\" (UniqueName: \"kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn\") pod \"payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.609541 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.592223    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.696087 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.695999    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.696502 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.696470    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9fn\" (UniqueName: \"kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn\") pod \"payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.702688 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.698619    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.722981 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.722928    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9fn\" (UniqueName: \"kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn\") pod \"payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:26.829878 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.829842    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:26.927669 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:26.927615    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:27.121601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.121545    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:27.170300 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.170262    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 30 17:59:27.170672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.170603    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 30 17:59:27.230239 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:27.230201    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944f5a3a_2c51_4f25_80b6_665442097124.slice/crio-620c0f73e43e97e3eaf1a884861d325b085538d19120b1e8dd4d98a2535f6e92 WatchSource:0}: Error finding container 620c0f73e43e97e3eaf1a884861d325b085538d19120b1e8dd4d98a2535f6e92: Status 404 returned error can't find the container with id 620c0f73e43e97e3eaf1a884861d325b085538d19120b1e8dd4d98a2535f6e92
Oct 30 17:59:27.233207 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.233175    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"]
Oct 30 17:59:27.403499 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.403401    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"]
Oct 30 17:59:27.895381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.895338    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548" event={"ID":"944f5a3a-2c51-4f25-80b6-665442097124","Type":"ContainerStarted","Data":"6f10493a431c78aa8799083aa58fff035d82593edb22b28ae264a4dfacb925f5"}
Oct 30 17:59:27.895601 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.895408    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548" event={"ID":"944f5a3a-2c51-4f25-80b6-665442097124","Type":"ContainerStarted","Data":"620c0f73e43e97e3eaf1a884861d325b085538d19120b1e8dd4d98a2535f6e92"}
Oct 30 17:59:27.895740 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.895621    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:27.898000 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.897976    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7" event={"ID":"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3","Type":"ContainerStarted","Data":"3f26720898fe478046d17d29342b817b663ff9ad8844116fb173e959d66ee6ad"}
Oct 30 17:59:27.898138 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.898022    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7" event={"ID":"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3","Type":"ContainerStarted","Data":"61600b2fb81a55464df73f92abb01e3db6723bc9b09bdf5ebd3e65acbcc49835"}
Oct 30 17:59:27.899145 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.899125    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:27.936720 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.936662    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548" podStartSLOduration=1.936630396 podStartE2EDuration="1.936630396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:27.935279193 +0000 UTC m=+2773.697323007" watchObservedRunningTime="2026-10-30 17:59:27.936630396 +0000 UTC m=+2773.698674209"
Oct 30 17:59:27.977667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:27.976706    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7" podStartSLOduration=1.976690006 podStartE2EDuration="1.976690006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:27.974832206 +0000 UTC m=+2773.736876020" watchObservedRunningTime="2026-10-30 17:59:27.976690006 +0000 UTC m=+2773.738733820"
Oct 30 17:59:28.212722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:28.212365    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 30 17:59:28.212722 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:28.212420    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 30 17:59:28.328736 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:28.328705    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:59:29.013521 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:29.013491    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:29.180699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:29.177790    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 30 17:59:29.180699 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:29.177850    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 30 17:59:29.695515 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:29.692053    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:30.167697 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:30.167635    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 30 17:59:30.168088 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:30.167724    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 30 17:59:30.754944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:30.754905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:31.174390 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.174359    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:59:31.175959 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.175935    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 17:59:31.967485 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.967446    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"]
Oct 30 17:59:31.975500 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.975473    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:31.989337 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.989311    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"]
Oct 30 17:59:31.996721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.996690    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:59:31.996865 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:31.996787    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:59:32.015242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.015050    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjf2\" (UniqueName: \"kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.015242 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.015177    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.015417 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.015246    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.116519 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.116486    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjf2\" (UniqueName: \"kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.116754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.116567    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.116754 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.116608    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.117370 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.117344    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.119175 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.119154    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.140128 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.140095    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjf2\" (UniqueName: \"kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2\") pod \"maas-api-revoke-sub-c8dcd3e3-tbz2d\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.301825 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.301739    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:32.546411 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.546378    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:32.615093 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:32.615061    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd400f341_6511_4e7e_b5bf_dce70dc44bba.slice/crio-49e3b24ad4de41ad2210607acbae300ae5d2b7f3c9ffb8b0f2a03b781a273a75 WatchSource:0}: Error finding container 49e3b24ad4de41ad2210607acbae300ae5d2b7f3c9ffb8b0f2a03b781a273a75: Status 404 returned error can't find the container with id 49e3b24ad4de41ad2210607acbae300ae5d2b7f3c9ffb8b0f2a03b781a273a75
Oct 30 17:59:32.616824 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.616799    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"]
Oct 30 17:59:32.941471 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:32.941438    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d" event={"ID":"d400f341-6511-4e7e-b5bf-dce70dc44bba","Type":"ContainerStarted","Data":"49e3b24ad4de41ad2210607acbae300ae5d2b7f3c9ffb8b0f2a03b781a273a75"}
Oct 30 17:59:33.955252 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:33.955216    5534 generic.go:356] "Generic (PLEG): container finished" podID="d400f341-6511-4e7e-b5bf-dce70dc44bba" containerID="ce825ac94316fef0ea77e9e8280885148b55466a84ed2267d10618cbd6d56df0" exitCode=0
Oct 30 17:59:33.955670 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:33.955491    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d" event={"ID":"d400f341-6511-4e7e-b5bf-dce70dc44bba","Type":"ContainerDied","Data":"ce825ac94316fef0ea77e9e8280885148b55466a84ed2267d10618cbd6d56df0"}
Oct 30 17:59:34.308241 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:34.308150    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:35.149995 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.149971    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:35.272446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.272358    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca\") pod \"d400f341-6511-4e7e-b5bf-dce70dc44bba\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") "
Oct 30 17:59:35.272446 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.272434    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca\") pod \"d400f341-6511-4e7e-b5bf-dce70dc44bba\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") "
Oct 30 17:59:35.272718 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.272514    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2\" (UniqueName: \"kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2\") pod \"d400f341-6511-4e7e-b5bf-dce70dc44bba\" (UID: \"d400f341-6511-4e7e-b5bf-dce70dc44bba\") "
Oct 30 17:59:35.272819 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.272790    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca" pod "d400f341-6511-4e7e-b5bf-dce70dc44bba" (UID: "d400f341-6511-4e7e-b5bf-dce70dc44bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:35.273617 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.273591    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:35.274873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.274834    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca" pod "d400f341-6511-4e7e-b5bf-dce70dc44bba" (UID: "d400f341-6511-4e7e-b5bf-dce70dc44bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:59:35.275013 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.274989    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2" pod "d400f341-6511-4e7e-b5bf-dce70dc44bba" (UID: "d400f341-6511-4e7e-b5bf-dce70dc44bba"). InnerVolumeSpecName "kube-api-access-sfjf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:35.374656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.374595    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d400f341-6511-4e7e-b5bf-dce70dc44bba-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:35.374656 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.374655    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjf2\" (UniqueName: \"kubernetes.io/projected/d400f341-6511-4e7e-b5bf-dce70dc44bba-kube-api-access-sfjf2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:35.977488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.977454    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"
Oct 30 17:59:35.977488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.977476    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d" event={"ID":"d400f341-6511-4e7e-b5bf-dce70dc44bba","Type":"ContainerDied","Data":"49e3b24ad4de41ad2210607acbae300ae5d2b7f3c9ffb8b0f2a03b781a273a75"}
Oct 30 17:59:35.977794 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.977571    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e3b24ad4de41ad2210607acbae300ae5d2b7f3c9ffb8b0f2a03b781a273a75"
Oct 30 17:59:35.991173 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:35.991143    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:38.411746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:38.411709    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:38.912156 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:38.912126    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 17:59:39.919098 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:39.919066    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 17:59:40.095301 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:40.095248    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:41.702944 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:41.702770    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:43.993342 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:43.993304    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:45.248840 ip-10-0-135-170 kubenswrapper[5534]: E1030 17:59:45.248804    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 17:59:45.548279 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.548196    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"]
Oct 30 17:59:45.549064 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.549047    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d400f341-6511-4e7e-b5bf-dce70dc44bba" containerName="revoke-keys"
Oct 30 17:59:45.549132 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.549066    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d400f341-6511-4e7e-b5bf-dce70dc44bba" containerName="revoke-keys"
Oct 30 17:59:45.549209 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.549200    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d400f341-6511-4e7e-b5bf-dce70dc44bba" containerName="revoke-keys"
Oct 30 17:59:45.553852 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.553828    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.573484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.572732    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"]
Oct 30 17:59:45.579874 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.579855    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:59:45.580070 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.579938    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:59:45.620928 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.620892    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr8x\" (UniqueName: \"kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.621361 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.621337    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.621564 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.621545    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.723672 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.723611    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr8x\" (UniqueName: \"kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.723907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.723787    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.723907 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.723812    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.724631 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.724604    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.726888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.726855    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.748062 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.747988    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr8x\" (UniqueName: \"kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x\") pod \"maas-api-revoke-sub-8099e35f-fnlcv\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:45.872312 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:45.872275    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:46.127991 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:46.127910    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:46.207194 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:46.207169    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"]
Oct 30 17:59:46.209270 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:46.209233    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6f8544_9941_48cd_942d_7d758c85da06.slice/crio-4965cc76bcf3355137d68c6755992841a3946c78e776497f955c417bbc4ba8a8 WatchSource:0}: Error finding container 4965cc76bcf3355137d68c6755992841a3946c78e776497f955c417bbc4ba8a8: Status 404 returned error can't find the container with id 4965cc76bcf3355137d68c6755992841a3946c78e776497f955c417bbc4ba8a8
Oct 30 17:59:46.871219 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:46.871180    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:46.997212 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:46.997179    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:47.137303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:47.137214    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:47.147100 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:47.147056    5534 generic.go:356] "Generic (PLEG): container finished" podID="5a6f8544-9941-48cd-942d-7d758c85da06" containerID="d5e5ed6ca68db28c7d25119373641b2ab797bd227da5525c4f06e9bfc2f967e1" exitCode=0
Oct 30 17:59:47.147308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:47.147143    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv" event={"ID":"5a6f8544-9941-48cd-942d-7d758c85da06","Type":"ContainerDied","Data":"d5e5ed6ca68db28c7d25119373641b2ab797bd227da5525c4f06e9bfc2f967e1"}
Oct 30 17:59:47.147308 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:47.147217    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv" event={"ID":"5a6f8544-9941-48cd-942d-7d758c85da06","Type":"ContainerStarted","Data":"4965cc76bcf3355137d68c6755992841a3946c78e776497f955c417bbc4ba8a8"}
Oct 30 17:59:47.847303 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:47.847268    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:47.966887 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:47.966855    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:48.339205 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.339180    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:48.486571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.486484    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca\") pod \"5a6f8544-9941-48cd-942d-7d758c85da06\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") "
Oct 30 17:59:48.486571 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.486550    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x\" (UniqueName: \"kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x\") pod \"5a6f8544-9941-48cd-942d-7d758c85da06\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") "
Oct 30 17:59:48.486873 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.486675    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca\") pod \"5a6f8544-9941-48cd-942d-7d758c85da06\" (UID: \"5a6f8544-9941-48cd-942d-7d758c85da06\") "
Oct 30 17:59:48.486985 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.486958    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca" pod "5a6f8544-9941-48cd-942d-7d758c85da06" (UID: "5a6f8544-9941-48cd-942d-7d758c85da06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:48.488015 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.487980    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:48.489213 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.489185    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca" pod "5a6f8544-9941-48cd-942d-7d758c85da06" (UID: "5a6f8544-9941-48cd-942d-7d758c85da06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:59:48.489484 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.489460    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x" pod "5a6f8544-9941-48cd-942d-7d758c85da06" (UID: "5a6f8544-9941-48cd-942d-7d758c85da06"). InnerVolumeSpecName "kube-api-access-dtr8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:48.589478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.589435    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtr8x\" (UniqueName: \"kubernetes.io/projected/5a6f8544-9941-48cd-942d-7d758c85da06-kube-api-access-dtr8x\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:48.589478 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:48.589477    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8544-9941-48cd-942d-7d758c85da06-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:49.167536 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:49.167508    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"
Oct 30 17:59:49.168063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:49.167543    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv" event={"ID":"5a6f8544-9941-48cd-942d-7d758c85da06","Type":"ContainerDied","Data":"4965cc76bcf3355137d68c6755992841a3946c78e776497f955c417bbc4ba8a8"}
Oct 30 17:59:49.168063 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:49.167601    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4965cc76bcf3355137d68c6755992841a3946c78e776497f955c417bbc4ba8a8"
Oct 30 17:59:49.768271 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:49.768234    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:51.363692 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.363630    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"]
Oct 30 17:59:51.364838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.364812    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5a6f8544-9941-48cd-942d-7d758c85da06" containerName="revoke-keys"
Oct 30 17:59:51.364838 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.364833    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6f8544-9941-48cd-942d-7d758c85da06" containerName="revoke-keys"
Oct 30 17:59:51.365012 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.364978    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6f8544-9941-48cd-942d-7d758c85da06" containerName="revoke-keys"
Oct 30 17:59:51.369160 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.369140    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.397192 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.397167    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fab25f-openshift-default-dockercfg-2rfnw\""
Oct 30 17:59:51.403525 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.403500    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"]
Oct 30 17:59:51.534052 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.534022    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:51.539103 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539082    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539191 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539112    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539286 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539267    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539440 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539423    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539530 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539505    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539689 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539592    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539696    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539746 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539724    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwv92\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.539847 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.539793    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641147    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641208    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641238 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641234    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641299    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641538 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641365    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwv92\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641709 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641682    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641805 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641780    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641808    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.641879 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.641830    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.642294 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.642273    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.642381 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.642073    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.642456 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.642440    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.642574 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.642555    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.643049 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.643026    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.643776 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.643757    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.644083 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.644065    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.664831 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.664802    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwv92\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.664955 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.664933    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:51.687110 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:51.687082    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:52.023414 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.023386    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"]
Oct 30 17:59:52.023542 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:52.023481    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22b1f6e_24fa_437f_809d_446fbf9f894d.slice/crio-aed327488548104cc365fb3fc71efd6827f0706378c5ef7264e37b69c0d83471 WatchSource:0}: Error finding container aed327488548104cc365fb3fc71efd6827f0706378c5ef7264e37b69c0d83471: Status 404 returned error can't find the container with id aed327488548104cc365fb3fc71efd6827f0706378c5ef7264e37b69c0d83471
Oct 30 17:59:52.026348 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.026317    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:52.026420 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.026387    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:52.026463 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.026418    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 17:59:52.202630 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.202583    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" event={"ID":"f22b1f6e-24fa-437f-809d-446fbf9f894d","Type":"ContainerStarted","Data":"505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da"}
Oct 30 17:59:52.202888 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.202694    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" event={"ID":"f22b1f6e-24fa-437f-809d-446fbf9f894d","Type":"ContainerStarted","Data":"aed327488548104cc365fb3fc71efd6827f0706378c5ef7264e37b69c0d83471"}
Oct 30 17:59:52.252391 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.252334    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podStartSLOduration=1.252315225 podStartE2EDuration="1.252315225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:52.250465801 +0000 UTC m=+2798.012509615" watchObservedRunningTime="2026-10-30 17:59:52.252315225 +0000 UTC m=+2798.014359040"
Oct 30 17:59:52.688179 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:52.688145    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:53.181667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:53.181217    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:53.301507 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:53.301474    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:53.689946 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:53.689906    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 17:59:53.690483 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:53.689966    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 17:59:54.691131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:54.691036    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 17:59:54.691131 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:54.691091    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 17:59:55.041453 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.041367    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:55.688367 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.688330    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 17:59:55.688778 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.688743    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 17:59:55.889741 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.889697    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"]
Oct 30 17:59:55.913203 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.912531    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:55.925095 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.925062    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"]
Oct 30 17:59:55.932866 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.932840    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:59:55.933008 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.932965    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 17:59:55.934901 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.934871    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsh2\" (UniqueName: \"kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:55.935089 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.935070    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:55.935230 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:55.935213    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.037229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.037136    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsh2\" (UniqueName: \"kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.037229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.037178    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.037229 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.037208    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.038270 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.038243    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.040176 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.040154    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.091023 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.090991    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsh2\" (UniqueName: \"kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2\") pod \"maas-api-revoke-sub-196e5b96-v7qm8\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.238582 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.238554    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:56.601988 ip-10-0-135-170 kubenswrapper[5534]: W1030 17:59:56.601184    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c70b37_f87a_4b3b_ac44_d9d075a65ffd.slice/crio-493dfa20dec67d061237dbfde4cdfd34149c0d5edb234d9be532ba5be98c5848 WatchSource:0}: Error finding container 493dfa20dec67d061237dbfde4cdfd34149c0d5edb234d9be532ba5be98c5848: Status 404 returned error can't find the container with id 493dfa20dec67d061237dbfde4cdfd34149c0d5edb234d9be532ba5be98c5848
Oct 30 17:59:56.601988 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.601824    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"]
Oct 30 17:59:56.691667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.691550    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 17:59:56.691667 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.691614    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 17:59:56.979668 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:56.976551    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:57.243613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:57.243528    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8" event={"ID":"72c70b37-f87a-4b3b-ac44-d9d075a65ffd","Type":"ContainerStarted","Data":"66ec69abad22e2143cc6df2b80434216223246f069e53008f65c166e39a93407"}
Oct 30 17:59:57.243613 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:57.243583    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8" event={"ID":"72c70b37-f87a-4b3b-ac44-d9d075a65ffd","Type":"ContainerStarted","Data":"493dfa20dec67d061237dbfde4cdfd34149c0d5edb234d9be532ba5be98c5848"}
Oct 30 17:59:57.295569 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:57.295517    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8" podStartSLOduration=2.2955023629999998 podStartE2EDuration="2.295502363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:57.290131001 +0000 UTC m=+2803.052174815" watchObservedRunningTime="2026-10-30 17:59:57.295502363 +0000 UTC m=+2803.057546176"
Oct 30 17:59:57.688401 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:57.688367    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 17:59:57.688721 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:57.688691    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 17:59:58.256973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:58.256920    5534 generic.go:356] "Generic (PLEG): container finished" podID="72c70b37-f87a-4b3b-ac44-d9d075a65ffd" containerID="66ec69abad22e2143cc6df2b80434216223246f069e53008f65c166e39a93407" exitCode=0
Oct 30 17:59:58.256973 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:58.256965    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8" event={"ID":"72c70b37-f87a-4b3b-ac44-d9d075a65ffd","Type":"ContainerDied","Data":"66ec69abad22e2143cc6df2b80434216223246f069e53008f65c166e39a93407"}
Oct 30 17:59:58.694956 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:58.694928    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:58.695514 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:58.695498    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:58.696339 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:58.696312    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 17:59:58.704488 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:58.704463    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 17:59:59.444139 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.444115    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 17:59:59.509868 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.509829    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca\") pod \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") "
Oct 30 17:59:59.510127 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.510101    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca\") pod \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") "
Oct 30 17:59:59.510275 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.510195    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2\" (UniqueName: \"kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2\") pod \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\" (UID: \"72c70b37-f87a-4b3b-ac44-d9d075a65ffd\") "
Oct 30 17:59:59.510473 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.510446    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca" pod "72c70b37-f87a-4b3b-ac44-d9d075a65ffd" (UID: "72c70b37-f87a-4b3b-ac44-d9d075a65ffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:59:59.510933 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.510878    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:59.512587 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.512563    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca" pod "72c70b37-f87a-4b3b-ac44-d9d075a65ffd" (UID: "72c70b37-f87a-4b3b-ac44-d9d075a65ffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:59:59.512947 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.512925    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2" pod "72c70b37-f87a-4b3b-ac44-d9d075a65ffd" (UID: "72c70b37-f87a-4b3b-ac44-d9d075a65ffd"). InnerVolumeSpecName "kube-api-access-xgsh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:59:59.612358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.612323    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgsh2\" (UniqueName: \"kubernetes.io/projected/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-kube-api-access-xgsh2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 17:59:59.612358 ip-10-0-135-170 kubenswrapper[5534]: I1030 17:59:59.612356    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c70b37-f87a-4b3b-ac44-d9d075a65ffd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:00.217984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.217951    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"]
Oct 30 18:00:00.219039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.219011    5534 cpu_manager.go:414] "containerMap: removing container" podUID="72c70b37-f87a-4b3b-ac44-d9d075a65ffd" containerName="revoke-keys"
Oct 30 18:00:00.219039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.219038    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c70b37-f87a-4b3b-ac44-d9d075a65ffd" containerName="revoke-keys"
Oct 30 18:00:00.219519 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.219258    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c70b37-f87a-4b3b-ac44-d9d075a65ffd" containerName="revoke-keys"
Oct 30 18:00:00.223941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.223917    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"]
Oct 30 18:00:00.224119 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.224093    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.229324 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.229295    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"]
Oct 30 18:00:00.229463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.229444    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.234739 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.234715    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"]
Oct 30 18:00:00.234885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.234848    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.241318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.241294    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.241501 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.241483    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"]
Oct 30 18:00:00.247866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.247549    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.261712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.251712    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"]
Oct 30 18:00:00.295782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.287831    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"]
Oct 30 18:00:00.302665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.296280    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"]
Oct 30 18:00:00.306904 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.306871    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"]
Oct 30 18:00:00.313046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.312888    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.314443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.314417    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"]
Oct 30 18:00:00.317696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.316781    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"
Oct 30 18:00:00.317696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.316777    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8" event={"ID":"72c70b37-f87a-4b3b-ac44-d9d075a65ffd","Type":"ContainerDied","Data":"493dfa20dec67d061237dbfde4cdfd34149c0d5edb234d9be532ba5be98c5848"}
Oct 30 18:00:00.317696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.316894    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493dfa20dec67d061237dbfde4cdfd34149c0d5edb234d9be532ba5be98c5848"
Oct 30 18:00:00.321543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321516    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg9m\" (UniqueName: \"kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.321728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321562    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.321728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321660    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrcq\" (UniqueName: \"kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.321728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321690    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.321923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321765    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52j8\" (UniqueName: \"kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.321923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321830    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.322043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321929    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7jp\" (UniqueName: \"kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.322043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321966    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgnqh\" (UniqueName: \"kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.322043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.321996    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.322992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.322151    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.354728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.348807    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"]
Oct 30 18:00:00.354728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.355070    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"]
Oct 30 18:00:00.392224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.384117    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"]
Oct 30 18:00:00.394635 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.394599    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"]
Oct 30 18:00:00.394923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.394905    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.410935 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.408481    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"]
Oct 30 18:00:00.410935 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.408595    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.421851    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"]
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.423990    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7jp\" (UniqueName: \"kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424045    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgnqh\" (UniqueName: \"kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424079    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424362    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424500    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg9m\" (UniqueName: \"kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424533    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424626    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrcq\" (UniqueName: \"kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424685    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424744    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424803    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsx8\" (UniqueName: \"kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424884    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52j8\" (UniqueName: \"kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.424925    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.425162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.425090    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.426183 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.425968    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.426183 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.426063    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.427187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.426852    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.427187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.427004    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.427187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.427073    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"]
Oct 30 18:00:00.436608 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.436463    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"]
Oct 30 18:00:00.436608 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.436577    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.444747 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.443461    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.448395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.448363    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgnqh\" (UniqueName: \"kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.460061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.451207    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7jp\" (UniqueName: \"kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.460061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.451260    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"]
Oct 30 18:00:00.460061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.457343    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"]
Oct 30 18:00:00.463852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.463823    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.467430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.467406    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"]
Oct 30 18:00:00.475832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.475763    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrcq\" (UniqueName: \"kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.477546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.477516    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52j8\" (UniqueName: \"kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8\") pod \"maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.481277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.481251    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg9m\" (UniqueName: \"kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.484378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.484355    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"]
Oct 30 18:00:00.527743 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.527703    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889720-wl9fr\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.527909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.527799    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.528451 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528425    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vr9\" (UniqueName: \"kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.528531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528515    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.528601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528556    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.528675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528620    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.528735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528690    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgl4k\" (UniqueName: \"kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k\") pod \"maas-api-key-cleanup-29889720-wl9fr\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.528735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528724    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfv9\" (UniqueName: \"kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.528867 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528773    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6htp\" (UniqueName: \"kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp\") pod \"maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.528867 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528801    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsx8\" (UniqueName: \"kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.528962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528879    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd854\" (UniqueName: \"kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.528962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.528921    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.530027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.529998    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.543607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.543581    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:00.550203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.550181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:00.553961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.553933    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsx8\" (UniqueName: \"kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8\") pod \"maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.619619 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.619326    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.638923    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.638932    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vr9\" (UniqueName: \"kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639268    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639290    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639332    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgl4k\" (UniqueName: \"kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k\") pod \"maas-api-key-cleanup-29889720-wl9fr\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639350    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfv9\" (UniqueName: \"kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639376    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6htp\" (UniqueName: \"kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp\") pod \"maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639425    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd854\" (UniqueName: \"kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639448    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639496    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889720-wl9fr\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.640139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.639533    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.649671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.641827    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.649671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.642426    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.649671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.643025    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.649671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.644397    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.649671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.645234    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889720-wl9fr\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.665253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.663604    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.674252    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vr9\" (UniqueName: \"kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.680402    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfv9\" (UniqueName: \"kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.682131    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6htp\" (UniqueName: \"kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp\") pod \"maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.685931    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.711471    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgl4k\" (UniqueName: \"kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k\") pod \"maas-api-key-cleanup-29889720-wl9fr\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.711492    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd854\" (UniqueName: \"kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:00.730576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.724344    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:00.739827 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.738293    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:00.747315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.742956    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:00.760816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.757279    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:00.913103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.913071    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:00.913949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.913921    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:00.914615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:00.914597    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:01.166845 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.166818    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"]
Oct 30 18:00:01.332291 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.332201    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerStarted","Data":"d23ceca6b7414ca5a8032e592b0b1558aeb6cab77e17d4935a9b406366fa600f"}
Oct 30 18:00:01.373715 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.373682    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:01.419446 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:01.419416    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77285cce_053d_4ea7_85d7_f0d4b38e7aa1.slice/crio-86a02c7b88605c56b11b6f523dcc5383f3e5adbc080a86d1e23e673c56c6bd58 WatchSource:0}: Error finding container 86a02c7b88605c56b11b6f523dcc5383f3e5adbc080a86d1e23e673c56c6bd58: Status 404 returned error can't find the container with id 86a02c7b88605c56b11b6f523dcc5383f3e5adbc080a86d1e23e673c56c6bd58
Oct 30 18:00:01.419606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.419487    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"]
Oct 30 18:00:01.450247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.450225    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"]
Oct 30 18:00:01.453067 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:01.453032    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac0500a_7c7b_4f5e_94ec_8ae263349221.slice/crio-23fd2086849a808903338047dbdc32d0163e11f2775ba453b0fd6f5b78d17b28 WatchSource:0}: Error finding container 23fd2086849a808903338047dbdc32d0163e11f2775ba453b0fd6f5b78d17b28: Status 404 returned error can't find the container with id 23fd2086849a808903338047dbdc32d0163e11f2775ba453b0fd6f5b78d17b28
Oct 30 18:00:01.458058 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.458014    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"]
Oct 30 18:00:01.518133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.518103    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"]
Oct 30 18:00:01.524184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.524153    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"]
Oct 30 18:00:01.547541 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:01.547491    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7783598_4314_49e8_9915_da38199cca33.slice/crio-3cfa459ebf4c4112271e467e25bc44d8e56740246f67bf3be1699cb64dc7e1fd WatchSource:0}: Error finding container 3cfa459ebf4c4112271e467e25bc44d8e56740246f67bf3be1699cb64dc7e1fd: Status 404 returned error can't find the container with id 3cfa459ebf4c4112271e467e25bc44d8e56740246f67bf3be1699cb64dc7e1fd
Oct 30 18:00:01.569865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.558032    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"]
Oct 30 18:00:01.576036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.575986    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"]
Oct 30 18:00:01.592260 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.592225    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"]
Oct 30 18:00:01.598752 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:01.598721    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3e5eed_bef2_4406_9baf_2d1008aafac9.slice/crio-a4ba52f7d26c33240d9443032007677277518bba7d800d29a864447d06eca872 WatchSource:0}: Error finding container a4ba52f7d26c33240d9443032007677277518bba7d800d29a864447d06eca872: Status 404 returned error can't find the container with id a4ba52f7d26c33240d9443032007677277518bba7d800d29a864447d06eca872
Oct 30 18:00:01.636658 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.636618    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"]
Oct 30 18:00:01.646786 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:01.646754    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7532b0_bf21_420f_a014_d4c00efc644d.slice/crio-974ab774cdb9d71c427f90700897d0d0ae70877f80b3e0d14b714c8fa51cc0a8 WatchSource:0}: Error finding container 974ab774cdb9d71c427f90700897d0d0ae70877f80b3e0d14b714c8fa51cc0a8: Status 404 returned error can't find the container with id 974ab774cdb9d71c427f90700897d0d0ae70877f80b3e0d14b714c8fa51cc0a8
Oct 30 18:00:01.671443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.671411    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"]
Oct 30 18:00:01.677929 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:01.677876    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1e8336_6765_4656_a1d3_28fead2c7d26.slice/crio-8ef42b24968a3b072c41901c799f1beb91c60343b004a9b7ec926dd65b7cd75d WatchSource:0}: Error finding container 8ef42b24968a3b072c41901c799f1beb91c60343b004a9b7ec926dd65b7cd75d: Status 404 returned error can't find the container with id 8ef42b24968a3b072c41901c799f1beb91c60343b004a9b7ec926dd65b7cd75d
Oct 30 18:00:01.956072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:01.956040    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:02.169017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.168913    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:02.345483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.345314    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerStarted","Data":"ccb71a81b343ba4cb25176bfc1bf890bd276d127a41b699a8df658f4bfd08727"}
Oct 30 18:00:02.346085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.345943    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerStarted","Data":"974ab774cdb9d71c427f90700897d0d0ae70877f80b3e0d14b714c8fa51cc0a8"}
Oct 30 18:00:02.574898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.574802    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" podStartSLOduration=2.574779659 podStartE2EDuration="2.574779659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.568297823 +0000 UTC m=+2808.330341639" watchObservedRunningTime="2026-10-30 18:00:02.574779659 +0000 UTC m=+2808.336823499"
Oct 30 18:00:02.614450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.614413    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerStarted","Data":"31bb51a01bca79c1d996bcb89a974579650c07f5bf4947f0e8f38b61b883f47a"}
Oct 30 18:00:02.614583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.614490    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerStarted","Data":"be848991466a226170fdf46cb35f7861d51f15c90af92ff92d8ea893042c0bfc"}
Oct 30 18:00:02.614583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.614517    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerStarted","Data":"436a81c74eed9258a3625186ce9c47156700bcc1918021fbb5efd6a85b6ce9c0"}
Oct 30 18:00:02.614583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.614543    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerStarted","Data":"90205a5d590c0f45c637a260c4ea86a81fc917be6470dc8dade85a6c06ca5299"}
Oct 30 18:00:02.614583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.614568    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerStarted","Data":"f6b52621d1baf321044cc9225be9648be16543d61cec1b2f605b4e4614e13a67"}
Oct 30 18:00:02.614854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.614590    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerStarted","Data":"23fd2086849a808903338047dbdc32d0163e11f2775ba453b0fd6f5b78d17b28"}
Oct 30 18:00:02.616450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.616422    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerStarted","Data":"9735cd13902121a36de04195f56d3a8f4495a4015888a88dea57ace07e9e1fd1"}
Oct 30 18:00:02.617713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.617511    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerStarted","Data":"8ef42b24968a3b072c41901c799f1beb91c60343b004a9b7ec926dd65b7cd75d"}
Oct 30 18:00:02.619347 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.619325    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" event={"ID":"d7783598-4314-49e8-9915-da38199cca33","Type":"ContainerStarted","Data":"3cfa459ebf4c4112271e467e25bc44d8e56740246f67bf3be1699cb64dc7e1fd"}
Oct 30 18:00:02.621546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.621522    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" event={"ID":"77285cce-053d-4ea7-85d7-f0d4b38e7aa1","Type":"ContainerStarted","Data":"86a02c7b88605c56b11b6f523dcc5383f3e5adbc080a86d1e23e673c56c6bd58"}
Oct 30 18:00:02.623848 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.623826    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerStarted","Data":"d08336fc1d924e0f458e4a548c09009b3e09c59a27e1db5fa61f53eec0364789"}
Oct 30 18:00:02.624396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.624373    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerStarted","Data":"a4ba52f7d26c33240d9443032007677277518bba7d800d29a864447d06eca872"}
Oct 30 18:00:02.639579 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.639544    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerStarted","Data":"16eab2a2bb05dd962377a5b0fa7393eb0e87f38b0d5360eb354d6e82abfe312c"}
Oct 30 18:00:02.639858 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.639839    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerStarted","Data":"ebb479d46c82f426c68b23c1ab01fc9a8bb5685807f483874a1bb81a7bcaae92"}
Oct 30 18:00:02.642596 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.642532    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" podStartSLOduration=2.642515308 podStartE2EDuration="2.642515308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.625127869 +0000 UTC m=+2808.387171682" watchObservedRunningTime="2026-10-30 18:00:02.642515308 +0000 UTC m=+2808.404559123"
Oct 30 18:00:02.700795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.698497    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"]
Oct 30 18:00:02.704985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.704960    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:02.707875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.707822    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" podStartSLOduration=2.707802169 podStartE2EDuration="2.707802169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.702691534 +0000 UTC m=+2808.464735345" watchObservedRunningTime="2026-10-30 18:00:02.707802169 +0000 UTC m=+2808.469845984"
Oct 30 18:00:02.716100 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.716070    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"]
Oct 30 18:00:02.728870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.728824    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-455e4a-dockercfg-zslsv\""
Oct 30 18:00:02.729628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.729606    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-455e4a\""
Oct 30 18:00:02.809144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.809037    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" podStartSLOduration=2.809023276 podStartE2EDuration="2.809023276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.804843166 +0000 UTC m=+2808.566886980" watchObservedRunningTime="2026-10-30 18:00:02.809023276 +0000 UTC m=+2808.571067089"
Oct 30 18:00:02.809635 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.809596    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" podStartSLOduration=2.809586758 podStartE2EDuration="2.809586758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.749095231 +0000 UTC m=+2808.511139046" watchObservedRunningTime="2026-10-30 18:00:02.809586758 +0000 UTC m=+2808.571630572"
Oct 30 18:00:02.846832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.846768    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" podStartSLOduration=2.846752493 podStartE2EDuration="2.846752493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.843554048 +0000 UTC m=+2808.605597862" watchObservedRunningTime="2026-10-30 18:00:02.846752493 +0000 UTC m=+2808.608796305"
Oct 30 18:00:02.877390 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.876774    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:02.877390 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.877052    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnfr\" (UniqueName: \"kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:02.979751 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.979722    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:02.979877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.979810    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnfr\" (UniqueName: \"kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:02.980446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:02.980426    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:03.054435 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.054062    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnfr\" (UniqueName: \"kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr\") pod \"payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:03.227747 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.227714    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"]
Oct 30 18:00:03.240014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.239828    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:03.326943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.326858    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:03.650206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.650168    5534 generic.go:356] "Generic (PLEG): container finished" podID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerID="f20512a7384d750c34058e72415b76d9221e5b8fa7f23d8890cc12868d588f4e" exitCode=60
Oct 30 18:00:03.650760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.650256    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerDied","Data":"f20512a7384d750c34058e72415b76d9221e5b8fa7f23d8890cc12868d588f4e"}
Oct 30 18:00:03.650760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.650573    5534 scope.go:122] "RemoveContainer" containerID="f20512a7384d750c34058e72415b76d9221e5b8fa7f23d8890cc12868d588f4e"
Oct 30 18:00:03.652344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.652316    5534 generic.go:356] "Generic (PLEG): container finished" podID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerID="f6b52621d1baf321044cc9225be9648be16543d61cec1b2f605b4e4614e13a67" exitCode=60
Oct 30 18:00:03.652490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.652410    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerDied","Data":"f6b52621d1baf321044cc9225be9648be16543d61cec1b2f605b4e4614e13a67"}
Oct 30 18:00:03.656448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.652805    5534 scope.go:122] "RemoveContainer" containerID="f6b52621d1baf321044cc9225be9648be16543d61cec1b2f605b4e4614e13a67"
Oct 30 18:00:03.656448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.655514    5534 generic.go:356] "Generic (PLEG): container finished" podID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerID="9735cd13902121a36de04195f56d3a8f4495a4015888a88dea57ace07e9e1fd1" exitCode=60
Oct 30 18:00:03.656448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.655910    5534 scope.go:122] "RemoveContainer" containerID="9735cd13902121a36de04195f56d3a8f4495a4015888a88dea57ace07e9e1fd1"
Oct 30 18:00:03.656448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.655940    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerDied","Data":"9735cd13902121a36de04195f56d3a8f4495a4015888a88dea57ace07e9e1fd1"}
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.658383    5534 generic.go:356] "Generic (PLEG): container finished" podID="d7783598-4314-49e8-9915-da38199cca33" containerID="3f9bbc68e7b3244c5cab25f6247c500af884f85a7841ab31322a4ef2d8aa5179" exitCode=60
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.658525    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" event={"ID":"d7783598-4314-49e8-9915-da38199cca33","Type":"ContainerDied","Data":"3f9bbc68e7b3244c5cab25f6247c500af884f85a7841ab31322a4ef2d8aa5179"}
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.658763    5534 scope.go:122] "RemoveContainer" containerID="3f9bbc68e7b3244c5cab25f6247c500af884f85a7841ab31322a4ef2d8aa5179"
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:03.660131    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8773137a_7b6e_4389_aa0f_08221a1f737c.slice/crio-f9e84b8ef59b0476d06f351a05ded622fe2757d2070c8cf5aa382d0df0731258 WatchSource:0}: Error finding container f9e84b8ef59b0476d06f351a05ded622fe2757d2070c8cf5aa382d0df0731258: Status 404 returned error can't find the container with id f9e84b8ef59b0476d06f351a05ded622fe2757d2070c8cf5aa382d0df0731258
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.660591    5534 generic.go:356] "Generic (PLEG): container finished" podID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerID="d93d45ec2f89ccaab29c4737aaadcd9a6aa09aae843f82869fc745812e1aeff6" exitCode=60
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.660754    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" event={"ID":"77285cce-053d-4ea7-85d7-f0d4b38e7aa1","Type":"ContainerDied","Data":"d93d45ec2f89ccaab29c4737aaadcd9a6aa09aae843f82869fc745812e1aeff6"}
Oct 30 18:00:03.661666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.661087    5534 scope.go:122] "RemoveContainer" containerID="d93d45ec2f89ccaab29c4737aaadcd9a6aa09aae843f82869fc745812e1aeff6"
Oct 30 18:00:03.665829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.664034    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"]
Oct 30 18:00:03.665829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.664140    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerID="d08336fc1d924e0f458e4a548c09009b3e09c59a27e1db5fa61f53eec0364789" exitCode=60
Oct 30 18:00:03.665829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.664221    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerDied","Data":"d08336fc1d924e0f458e4a548c09009b3e09c59a27e1db5fa61f53eec0364789"}
Oct 30 18:00:03.665829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.664426    5534 scope.go:122] "RemoveContainer" containerID="d08336fc1d924e0f458e4a548c09009b3e09c59a27e1db5fa61f53eec0364789"
Oct 30 18:00:03.667344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.666510    5534 generic.go:356] "Generic (PLEG): container finished" podID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerID="16eab2a2bb05dd962377a5b0fa7393eb0e87f38b0d5360eb354d6e82abfe312c" exitCode=60
Oct 30 18:00:03.667344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.666599    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerDied","Data":"16eab2a2bb05dd962377a5b0fa7393eb0e87f38b0d5360eb354d6e82abfe312c"}
Oct 30 18:00:03.667344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.666842    5534 scope.go:122] "RemoveContainer" containerID="16eab2a2bb05dd962377a5b0fa7393eb0e87f38b0d5360eb354d6e82abfe312c"
Oct 30 18:00:03.673151 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.673128    5534 generic.go:356] "Generic (PLEG): container finished" podID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerID="ccb71a81b343ba4cb25176bfc1bf890bd276d127a41b699a8df658f4bfd08727" exitCode=60
Oct 30 18:00:03.673258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.673217    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerDied","Data":"ccb71a81b343ba4cb25176bfc1bf890bd276d127a41b699a8df658f4bfd08727"}
Oct 30 18:00:03.673537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.673519    5534 scope.go:122] "RemoveContainer" containerID="ccb71a81b343ba4cb25176bfc1bf890bd276d127a41b699a8df658f4bfd08727"
Oct 30 18:00:03.675739 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.675720    5534 generic.go:356] "Generic (PLEG): container finished" podID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerID="4d6ad7c0fb5f432335765fb06c80c14749235f1296a56685c0b4c48186906657" exitCode=60
Oct 30 18:00:03.675826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.675806    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerDied","Data":"4d6ad7c0fb5f432335765fb06c80c14749235f1296a56685c0b4c48186906657"}
Oct 30 18:00:03.676042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.676027    5534 scope.go:122] "RemoveContainer" containerID="4d6ad7c0fb5f432335765fb06c80c14749235f1296a56685c0b4c48186906657"
Oct 30 18:00:03.682537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.682458    5534 generic.go:356] "Generic (PLEG): container finished" podID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerID="ace0394d65d4c4ca2fd0067fb6fffbb8287550cc9dbf5742a8877d5391261978" exitCode=60
Oct 30 18:00:03.682537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.682483    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerDied","Data":"ace0394d65d4c4ca2fd0067fb6fffbb8287550cc9dbf5742a8877d5391261978"}
Oct 30 18:00:03.683188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.683166    5534 scope.go:122] "RemoveContainer" containerID="ace0394d65d4c4ca2fd0067fb6fffbb8287550cc9dbf5742a8877d5391261978"
Oct 30 18:00:03.696751 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.696722    5534 generic.go:356] "Generic (PLEG): container finished" podID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerID="436a81c74eed9258a3625186ce9c47156700bcc1918021fbb5efd6a85b6ce9c0" exitCode=60
Oct 30 18:00:03.697212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.696982    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerDied","Data":"436a81c74eed9258a3625186ce9c47156700bcc1918021fbb5efd6a85b6ce9c0"}
Oct 30 18:00:03.697319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.697043    5534 scope.go:122] "RemoveContainer" containerID="436a81c74eed9258a3625186ce9c47156700bcc1918021fbb5efd6a85b6ce9c0"
Oct 30 18:00:03.697730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:03.697605    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" containerID="cri-o://4e42df9cded89e52f8a50c3c1d781e06bc5fc0d18fd7072334580cae637b71a6" gracePeriod=30
Oct 30 18:00:04.709737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.709695    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerStarted","Data":"1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb"}
Oct 30 18:00:04.712155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.712125    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerStarted","Data":"c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586"}
Oct 30 18:00:04.714370 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.714342    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerStarted","Data":"2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d"}
Oct 30 18:00:04.718249 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.718196    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" event={"ID":"d7783598-4314-49e8-9915-da38199cca33","Type":"ContainerStarted","Data":"1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f"}
Oct 30 18:00:04.720416 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.720387    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" event={"ID":"77285cce-053d-4ea7-85d7-f0d4b38e7aa1","Type":"ContainerStarted","Data":"84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7"}
Oct 30 18:00:04.722679 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.722637    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerStarted","Data":"42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3"}
Oct 30 18:00:04.724888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.724855    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerStarted","Data":"56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39"}
Oct 30 18:00:04.727584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.727554    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerStarted","Data":"83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1"}
Oct 30 18:00:04.730130 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.730103    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerStarted","Data":"3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12"}
Oct 30 18:00:04.732171 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.732149    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerStarted","Data":"b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0"}
Oct 30 18:00:04.734125 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.734100    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerStarted","Data":"3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee"}
Oct 30 18:00:04.735902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.735877    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b" event={"ID":"8773137a-7b6e-4389-aa0f-08221a1f737c","Type":"ContainerStarted","Data":"7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f"}
Oct 30 18:00:04.736016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.735918    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b" event={"ID":"8773137a-7b6e-4389-aa0f-08221a1f737c","Type":"ContainerStarted","Data":"f9e84b8ef59b0476d06f351a05ded622fe2757d2070c8cf5aa382d0df0731258"}
Oct 30 18:00:04.736016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.735972    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:04.783578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.783519    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" podStartSLOduration=4.783502717 podStartE2EDuration="4.783502717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:04.780203467 +0000 UTC m=+2810.542247281" watchObservedRunningTime="2026-10-30 18:00:04.783502717 +0000 UTC m=+2810.545546564"
Oct 30 18:00:04.960116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:04.957823    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" podStartSLOduration=4.95778752 podStartE2EDuration="4.95778752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:04.952253706 +0000 UTC m=+2810.714297531" watchObservedRunningTime="2026-10-30 18:00:04.95778752 +0000 UTC m=+2810.719831334"
Oct 30 18:00:05.076306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.076233    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" podStartSLOduration=5.076208418 podStartE2EDuration="5.076208418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:05.070476096 +0000 UTC m=+2810.832519911" watchObservedRunningTime="2026-10-30 18:00:05.076208418 +0000 UTC m=+2810.838252232"
Oct 30 18:00:05.162168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.162099    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" podStartSLOduration=5.162076607 podStartE2EDuration="5.162076607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:05.157310338 +0000 UTC m=+2810.919354152" watchObservedRunningTime="2026-10-30 18:00:05.162076607 +0000 UTC m=+2810.924120425"
Oct 30 18:00:05.201268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.201195    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b" podStartSLOduration=3.201180631 podStartE2EDuration="3.201180631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:05.200372691 +0000 UTC m=+2810.962416505" watchObservedRunningTime="2026-10-30 18:00:05.201180631 +0000 UTC m=+2810.963224445"
Oct 30 18:00:05.247428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.247364    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" podStartSLOduration=5.247347443 podStartE2EDuration="5.247347443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:05.242151636 +0000 UTC m=+2811.004195449" watchObservedRunningTime="2026-10-30 18:00:05.247347443 +0000 UTC m=+2811.009391255"
Oct 30 18:00:05.268334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.268299    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:05.747704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.747657    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerID="42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3" exitCode=60
Oct 30 18:00:05.748277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.747739    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerDied","Data":"42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3"}
Oct 30 18:00:05.748277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.747816    5534 scope.go:122] "RemoveContainer" containerID="d08336fc1d924e0f458e4a548c09009b3e09c59a27e1db5fa61f53eec0364789"
Oct 30 18:00:05.748277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.748203    5534 scope.go:122] "RemoveContainer" containerID="42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3"
Oct 30 18:00:05.748721 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.748694    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr_odh-ai-gateway-infra(2f3e5eed-bef2-4406-9baf-2d1008aafac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9"
Oct 30 18:00:05.750600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.750573    5534 generic.go:356] "Generic (PLEG): container finished" podID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerID="56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39" exitCode=60
Oct 30 18:00:05.750781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.750673    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerDied","Data":"56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39"}
Oct 30 18:00:05.751126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.751055    5534 scope.go:122] "RemoveContainer" containerID="56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39"
Oct 30 18:00:05.751466 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.751442    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh_odh-ai-gateway-infra(6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a"
Oct 30 18:00:05.753161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.753138    5534 generic.go:356] "Generic (PLEG): container finished" podID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerID="83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1" exitCode=60
Oct 30 18:00:05.753297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.753243    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerDied","Data":"83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1"}
Oct 30 18:00:05.753592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.753571    5534 scope.go:122] "RemoveContainer" containerID="83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1"
Oct 30 18:00:05.754058 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.754028    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs_odh-ai-gateway-infra(3f7532b0-bf21-420f-a014-d4c00efc644d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d"
Oct 30 18:00:05.755627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.755593    5534 generic.go:356] "Generic (PLEG): container finished" podID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerID="3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12" exitCode=60
Oct 30 18:00:05.755753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.755674    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerDied","Data":"3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12"}
Oct 30 18:00:05.756120 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.756094    5534 scope.go:122] "RemoveContainer" containerID="3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12"
Oct 30 18:00:05.756582 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.756540    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-wl9fr_odh-ai-gateway-infra(96cdda84-ee1f-43f0-83ef-b235c50d3f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85"
Oct 30 18:00:05.758468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.758441    5534 generic.go:356] "Generic (PLEG): container finished" podID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerID="b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0" exitCode=60
Oct 30 18:00:05.758571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.758551    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerDied","Data":"b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0"}
Oct 30 18:00:05.759024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.758996    5534 scope.go:122] "RemoveContainer" containerID="b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0"
Oct 30 18:00:05.759517 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.759482    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b_odh-ai-gateway-infra(e9cf8f81-e7d4-40b1-850d-6cc1126cb60e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e"
Oct 30 18:00:05.761045 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.761025    5534 generic.go:356] "Generic (PLEG): container finished" podID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerID="3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee" exitCode=60
Oct 30 18:00:05.761131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.761112    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerDied","Data":"3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee"}
Oct 30 18:00:05.761519 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.761499    5534 scope.go:122] "RemoveContainer" containerID="3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee"
Oct 30 18:00:05.762000 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.761945    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m_odh-ai-gateway-infra(fbe4fbe3-73ec-4d07-b82a-6d290ffa4952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952"
Oct 30 18:00:05.764063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.764036    5534 generic.go:356] "Generic (PLEG): container finished" podID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerID="1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb" exitCode=60
Oct 30 18:00:05.764164 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.764129    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerDied","Data":"1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb"}
Oct 30 18:00:05.764483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.764464    5534 scope.go:122] "RemoveContainer" containerID="1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb"
Oct 30 18:00:05.764955 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.764931    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45_odh-ai-gateway-infra(d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3"
Oct 30 18:00:05.766591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.766420    5534 generic.go:356] "Generic (PLEG): container finished" podID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerID="c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586" exitCode=60
Oct 30 18:00:05.766591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.766497    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerDied","Data":"c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586"}
Oct 30 18:00:05.766591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.766570    5534 scope.go:122] "RemoveContainer" containerID="16eab2a2bb05dd962377a5b0fa7393eb0e87f38b0d5360eb354d6e82abfe312c"
Oct 30 18:00:05.766861 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.766840    5534 scope.go:122] "RemoveContainer" containerID="c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586"
Oct 30 18:00:05.767268 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.767245    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm_odh-ai-gateway-infra(9ac0500a-7c7b-4f5e-94ec-8ae263349221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221"
Oct 30 18:00:05.769272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.769248    5534 generic.go:356] "Generic (PLEG): container finished" podID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerID="2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d" exitCode=60
Oct 30 18:00:05.769378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.769333    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerDied","Data":"2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d"}
Oct 30 18:00:05.769725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.769695    5534 scope.go:122] "RemoveContainer" containerID="2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d"
Oct 30 18:00:05.770146 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.770117    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg_odh-ai-gateway-infra(6f1e8336-6765-4656-a1d3-28fead2c7d26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26"
Oct 30 18:00:05.771985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.771958    5534 generic.go:356] "Generic (PLEG): container finished" podID="d7783598-4314-49e8-9915-da38199cca33" containerID="1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f" exitCode=60
Oct 30 18:00:05.772147 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.772122    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" event={"ID":"d7783598-4314-49e8-9915-da38199cca33","Type":"ContainerDied","Data":"1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f"}
Oct 30 18:00:05.772436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.772416    5534 scope.go:122] "RemoveContainer" containerID="1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f"
Oct 30 18:00:05.772867 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.772840    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2_odh-ai-gateway-infra(d7783598-4314-49e8-9915-da38199cca33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" podUID="d7783598-4314-49e8-9915-da38199cca33"
Oct 30 18:00:05.774942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.774916    5534 generic.go:356] "Generic (PLEG): container finished" podID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerID="84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7" exitCode=60
Oct 30 18:00:05.775074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.774994    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" event={"ID":"77285cce-053d-4ea7-85d7-f0d4b38e7aa1","Type":"ContainerDied","Data":"84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7"}
Oct 30 18:00:05.775410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.775392    5534 scope.go:122] "RemoveContainer" containerID="84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7"
Oct 30 18:00:05.775823 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:05.775801    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7_odh-ai-gateway-infra(77285cce-053d-4ea7-85d7-f0d4b38e7aa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1"
Oct 30 18:00:05.784936 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.784912    5534 scope.go:122] "RemoveContainer" containerID="ccb71a81b343ba4cb25176bfc1bf890bd276d127a41b699a8df658f4bfd08727"
Oct 30 18:00:05.802260 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.802238    5534 scope.go:122] "RemoveContainer" containerID="4d6ad7c0fb5f432335765fb06c80c14749235f1296a56685c0b4c48186906657"
Oct 30 18:00:05.821056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.821030    5534 scope.go:122] "RemoveContainer" containerID="ace0394d65d4c4ca2fd0067fb6fffbb8287550cc9dbf5742a8877d5391261978"
Oct 30 18:00:05.839364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.839332    5534 scope.go:122] "RemoveContainer" containerID="436a81c74eed9258a3625186ce9c47156700bcc1918021fbb5efd6a85b6ce9c0"
Oct 30 18:00:05.856368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.856347    5534 scope.go:122] "RemoveContainer" containerID="f20512a7384d750c34058e72415b76d9221e5b8fa7f23d8890cc12868d588f4e"
Oct 30 18:00:05.874588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.874564    5534 scope.go:122] "RemoveContainer" containerID="f6b52621d1baf321044cc9225be9648be16543d61cec1b2f605b4e4614e13a67"
Oct 30 18:00:05.894772 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.894740    5534 scope.go:122] "RemoveContainer" containerID="9735cd13902121a36de04195f56d3a8f4495a4015888a88dea57ace07e9e1fd1"
Oct 30 18:00:05.913118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.913089    5534 scope.go:122] "RemoveContainer" containerID="3f9bbc68e7b3244c5cab25f6247c500af884f85a7841ab31322a4ef2d8aa5179"
Oct 30 18:00:05.929885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:05.929863    5534 scope.go:122] "RemoveContainer" containerID="d93d45ec2f89ccaab29c4737aaadcd9a6aa09aae843f82869fc745812e1aeff6"
Oct 30 18:00:06.786038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.786008    5534 scope.go:122] "RemoveContainer" containerID="56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39"
Oct 30 18:00:06.786514 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.786463    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh_odh-ai-gateway-infra(6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a"
Oct 30 18:00:06.788193 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.788167    5534 scope.go:122] "RemoveContainer" containerID="83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1"
Oct 30 18:00:06.788590 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.788568    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs_odh-ai-gateway-infra(3f7532b0-bf21-420f-a014-d4c00efc644d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d"
Oct 30 18:00:06.790355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.790332    5534 scope.go:122] "RemoveContainer" containerID="3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12"
Oct 30 18:00:06.790760 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.790737    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-wl9fr_odh-ai-gateway-infra(96cdda84-ee1f-43f0-83ef-b235c50d3f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85"
Oct 30 18:00:06.792446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.792424    5534 scope.go:122] "RemoveContainer" containerID="b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0"
Oct 30 18:00:06.792835 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.792814    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b_odh-ai-gateway-infra(e9cf8f81-e7d4-40b1-850d-6cc1126cb60e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e"
Oct 30 18:00:06.794379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.794357    5534 scope.go:122] "RemoveContainer" containerID="3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee"
Oct 30 18:00:06.794756 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.794727    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m_odh-ai-gateway-infra(fbe4fbe3-73ec-4d07-b82a-6d290ffa4952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952"
Oct 30 18:00:06.796343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.796325    5534 scope.go:122] "RemoveContainer" containerID="1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb"
Oct 30 18:00:06.796697 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.796633    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45_odh-ai-gateway-infra(d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3"
Oct 30 18:00:06.798428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.798406    5534 scope.go:122] "RemoveContainer" containerID="c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586"
Oct 30 18:00:06.798790 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.798767    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm_odh-ai-gateway-infra(9ac0500a-7c7b-4f5e-94ec-8ae263349221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221"
Oct 30 18:00:06.800246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.800229    5534 scope.go:122] "RemoveContainer" containerID="2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d"
Oct 30 18:00:06.800541 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.800521    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg_odh-ai-gateway-infra(6f1e8336-6765-4656-a1d3-28fead2c7d26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26"
Oct 30 18:00:06.802092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.802071    5534 scope.go:122] "RemoveContainer" containerID="1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f"
Oct 30 18:00:06.802364 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.802346    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2_odh-ai-gateway-infra(d7783598-4314-49e8-9915-da38199cca33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" podUID="d7783598-4314-49e8-9915-da38199cca33"
Oct 30 18:00:06.804078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.804061    5534 scope.go:122] "RemoveContainer" containerID="84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7"
Oct 30 18:00:06.804349 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.804329    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7_odh-ai-gateway-infra(77285cce-053d-4ea7-85d7-f0d4b38e7aa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1"
Oct 30 18:00:06.805961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:06.805944    5534 scope.go:122] "RemoveContainer" containerID="42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3"
Oct 30 18:00:06.806212 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:06.806192    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr_odh-ai-gateway-infra(2f3e5eed-bef2-4406-9baf-2d1008aafac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9"
Oct 30 18:00:07.018530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:07.018494    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-g5nm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:00:07.018731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:07.018548    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:00:07.201989 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:07.201953    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:08.828458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:08.828410    5534 generic.go:356] "Generic (PLEG): container finished" podID="803484b3-0ab3-403f-9f7f-152d7da42061" containerID="4e42df9cded89e52f8a50c3c1d781e06bc5fc0d18fd7072334580cae637b71a6" exitCode=0
Oct 30 18:00:08.828948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:08.828453    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" event={"ID":"803484b3-0ab3-403f-9f7f-152d7da42061","Type":"ContainerDied","Data":"4e42df9cded89e52f8a50c3c1d781e06bc5fc0d18fd7072334580cae637b71a6"}
Oct 30 18:00:08.879376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:08.879325    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:09.057462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.057432    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 18:00:09.110621 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.110576    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.110849 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.110687    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.110893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.110863    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.110927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.110902    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.110980 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.110953    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.111035 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111023    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.111106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111050    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:09.111169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111080    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.111224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111194    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.111224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111209    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:09.111338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111236    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:09.111338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111313    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy\") pod \"803484b3-0ab3-403f-9f7f-152d7da42061\" (UID: \"803484b3-0ab3-403f-9f7f-152d7da42061\") "
Oct 30 18:00:09.111458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111351    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:09.111626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.111599    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:09.112079 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.112056    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.112200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.112080    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803484b3-0ab3-403f-9f7f-152d7da42061-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.112200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.112094    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.112200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.112107    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.112200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.112122    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.113600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.113566    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:09.113939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.113918    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "kube-api-access-qhn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:09.114056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.114035    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:09.114095 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.114048    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo" pod "803484b3-0ab3-403f-9f7f-152d7da42061" (UID: "803484b3-0ab3-403f-9f7f-152d7da42061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:00:09.213520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.213479    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803484b3-0ab3-403f-9f7f-152d7da42061-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.213520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.213518    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803484b3-0ab3-403f-9f7f-152d7da42061-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.213520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.213527    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.213793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.213536    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhn6p\" (UniqueName: \"kubernetes.io/projected/803484b3-0ab3-403f-9f7f-152d7da42061-kube-api-access-qhn6p\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:09.839948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.839915    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"
Oct 30 18:00:09.840418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.839946    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8" event={"ID":"803484b3-0ab3-403f-9f7f-152d7da42061","Type":"ContainerDied","Data":"a2619a6c72a05c36627aa08f067ed529c4bd6dffc49a4f7d8e82d61a1ba6c033"}
Oct 30 18:00:09.840418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.840011    5534 scope.go:122] "RemoveContainer" containerID="4e42df9cded89e52f8a50c3c1d781e06bc5fc0d18fd7072334580cae637b71a6"
Oct 30 18:00:09.909662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.909608    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"]
Oct 30 18:00:09.919038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:09.919006    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-g5nm8"]
Oct 30 18:00:10.375071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:10.375020    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" path="/var/lib/kubelet/pods/803484b3-0ab3-403f-9f7f-152d7da42061/volumes"
Oct 30 18:00:10.503027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:10.502992    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:11.845940 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:11.845898    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:12.246246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.246152    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"]
Oct 30 18:00:12.247062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.247045    5534 cpu_manager.go:414] "containerMap: removing container" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy"
Oct 30 18:00:12.247134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.247064    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy"
Oct 30 18:00:12.247216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.247207    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="803484b3-0ab3-403f-9f7f-152d7da42061" containerName="istio-proxy"
Oct 30 18:00:12.251223 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.251203    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.270877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.270811    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"]
Oct 30 18:00:12.277099 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.277065    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e927d7-serving-cert\""
Oct 30 18:00:12.357941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.357905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:12.377910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.377872    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.378085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.377963    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.378153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.378137    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.378253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.378240    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2ft\" (UniqueName: \"kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.477734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.477687    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:12.479309 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.479282    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.479487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.479399    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2ft\" (UniqueName: \"kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.479709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.479678    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.479869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.479734    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.480333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.480302    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.482091 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.482068    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.482269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.482252    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.526182 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.526065    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2ft\" (UniqueName: \"kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft\") pod \"maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.568614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.568579    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:12.592019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.591993    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:12.887942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.887799    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"]
Oct 30 18:00:12.891583 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:12.891552    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e5f9bc_0833_4071_bb36_941e32e98430.slice/crio-07b0654d9590ac545d6895e9345f693bf9979bb31963eb66cad7290386617f39 WatchSource:0}: Error finding container 07b0654d9590ac545d6895e9345f693bf9979bb31963eb66cad7290386617f39: Status 404 returned error can't find the container with id 07b0654d9590ac545d6895e9345f693bf9979bb31963eb66cad7290386617f39
Oct 30 18:00:12.893502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:12.893480    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:00:13.891504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:13.891466    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9" event={"ID":"d3e5f9bc-0833-4071-bb36-941e32e98430","Type":"ContainerStarted","Data":"426d272b5f757cc9a2f48e8a752d42125ebe6b17feeb6079cc1537f659a5f970"}
Oct 30 18:00:13.892049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:13.891536    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9" event={"ID":"d3e5f9bc-0833-4071-bb36-941e32e98430","Type":"ContainerStarted","Data":"07b0654d9590ac545d6895e9345f693bf9979bb31963eb66cad7290386617f39"}
Oct 30 18:00:13.892049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:13.891567    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:13.933575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:13.933520    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9" podStartSLOduration=1.545363718 podStartE2EDuration="1.933507232s" totalImagesPullingTime="388.143514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:13.932432655 +0000 UTC m=+2819.694476467" watchObservedRunningTime="2026-10-30 18:00:13.933507232 +0000 UTC m=+2819.695551044"
Oct 30 18:00:14.088205 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:14.088173    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:15.249623 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:15.249575    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:00:15.262366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.262332    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:15.362819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.362790    5534 scope.go:122] "RemoveContainer" containerID="2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d"
Oct 30 18:00:15.363172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.363150    5534 scope.go:122] "RemoveContainer" containerID="b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0"
Oct 30 18:00:15.758359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.758317    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:15.777846 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.777808    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:00:15.898184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.898137    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:15.912828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:15.912793    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerStarted","Data":"846948f95c3e10517b46c75d6f8d7441db874b4f068d909c6de57b011dbf4448"}
Oct 30 18:00:16.044263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.044232    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:16.166735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.166701    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:16.362847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.362817    5534 scope.go:122] "RemoveContainer" containerID="c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586"
Oct 30 18:00:16.363340 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.363006    5534 scope.go:122] "RemoveContainer" containerID="83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1"
Oct 30 18:00:16.363430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.363408    5534 scope.go:122] "RemoveContainer" containerID="3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12"
Oct 30 18:00:16.364157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.363733    5534 scope.go:122] "RemoveContainer" containerID="3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee"
Oct 30 18:00:16.923114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.923033    5534 generic.go:356] "Generic (PLEG): container finished" podID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerID="e6321ab510654fe29f2ebc679c672eab39da7e0bc4d8a478067eed1796bf9fc3" exitCode=60
Oct 30 18:00:16.923270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.923155    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerDied","Data":"e6321ab510654fe29f2ebc679c672eab39da7e0bc4d8a478067eed1796bf9fc3"}
Oct 30 18:00:16.923270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.923223    5534 scope.go:122] "RemoveContainer" containerID="2237c438b57275fa62bb3c788e067a885e2302e8116cf74a966163a279e97e1d"
Oct 30 18:00:16.923754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.923735    5534 scope.go:122] "RemoveContainer" containerID="e6321ab510654fe29f2ebc679c672eab39da7e0bc4d8a478067eed1796bf9fc3"
Oct 30 18:00:16.924159 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:16.924135    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg_odh-ai-gateway-infra(6f1e8336-6765-4656-a1d3-28fead2c7d26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26"
Oct 30 18:00:16.925507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.925483    5534 generic.go:356] "Generic (PLEG): container finished" podID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerID="846948f95c3e10517b46c75d6f8d7441db874b4f068d909c6de57b011dbf4448" exitCode=60
Oct 30 18:00:16.925628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.925546    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerDied","Data":"846948f95c3e10517b46c75d6f8d7441db874b4f068d909c6de57b011dbf4448"}
Oct 30 18:00:16.926011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.925987    5534 scope.go:122] "RemoveContainer" containerID="846948f95c3e10517b46c75d6f8d7441db874b4f068d909c6de57b011dbf4448"
Oct 30 18:00:16.926331 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:16.926304    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b_odh-ai-gateway-infra(e9cf8f81-e7d4-40b1-850d-6cc1126cb60e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e"
Oct 30 18:00:16.939248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.939201    5534 scope.go:122] "RemoveContainer" containerID="b4989e9f5dbb57abeb52be77e027effea0227c29480a193cd21ecd4baaff42a0"
Oct 30 18:00:16.983008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:16.982977    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"]
Oct 30 18:00:17.363064 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.363032    5534 scope.go:122] "RemoveContainer" containerID="84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7"
Oct 30 18:00:17.363623 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.363441    5534 scope.go:122] "RemoveContainer" containerID="56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39"
Oct 30 18:00:17.363623 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.363550    5534 scope.go:122] "RemoveContainer" containerID="1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f"
Oct 30 18:00:17.364255 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.364113    5534 scope.go:122] "RemoveContainer" containerID="42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3"
Oct 30 18:00:17.451517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.451490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:17.576041 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.575368    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:17.623738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.623631    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"]
Oct 30 18:00:17.630333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.630300    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:17.649132 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.649097    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"]
Oct 30 18:00:17.653606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.653556    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-584a12-dockercfg-jzcpq\""
Oct 30 18:00:17.704807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.693354    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-584a12\""
Oct 30 18:00:17.808946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.807966    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:17.808946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.808193    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn\") pod \"payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:17.912929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.912158    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn\") pod \"payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:17.912929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.912375    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:17.914612 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.914582    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:17.991563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:17.990231    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"]
Oct 30 18:00:18.012543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.012506    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerStarted","Data":"d0a825579b110248fb4cf40c0826169830763abf008af5b8cd217beb9e9db5d2"}
Oct 30 18:00:18.013436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.013410    5534 scope.go:122] "RemoveContainer" containerID="d0a825579b110248fb4cf40c0826169830763abf008af5b8cd217beb9e9db5d2"
Oct 30 18:00:18.014278 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:18.013748    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-wl9fr_odh-ai-gateway-infra(96cdda84-ee1f-43f0-83ef-b235c50d3f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85"
Oct 30 18:00:18.023111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.023076    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerStarted","Data":"59d37cdfa0236848901fd69fdc5d3bb6924acc6e633953d9150f153c867a3a3a"}
Oct 30 18:00:18.023590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.023572    5534 scope.go:122] "RemoveContainer" containerID="59d37cdfa0236848901fd69fdc5d3bb6924acc6e633953d9150f153c867a3a3a"
Oct 30 18:00:18.024045 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:18.024017    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m_odh-ai-gateway-infra(fbe4fbe3-73ec-4d07-b82a-6d290ffa4952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952"
Oct 30 18:00:18.026167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.026140    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerStarted","Data":"d02816cfc23bf87a39bea6dedea2b79ae49163b38945a4071e86cd65be497165"}
Oct 30 18:00:18.026699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.026581    5534 scope.go:122] "RemoveContainer" containerID="d02816cfc23bf87a39bea6dedea2b79ae49163b38945a4071e86cd65be497165"
Oct 30 18:00:18.027015 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:18.026995    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm_odh-ai-gateway-infra(9ac0500a-7c7b-4f5e-94ec-8ae263349221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221"
Oct 30 18:00:18.030834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.030811    5534 generic.go:356] "Generic (PLEG): container finished" podID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerID="d68fa9c2d891634aa6a62fa2c000f27d745c34d3f0be5a157684715876c0c537" exitCode=60
Oct 30 18:00:18.030983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.030968    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerDied","Data":"d68fa9c2d891634aa6a62fa2c000f27d745c34d3f0be5a157684715876c0c537"}
Oct 30 18:00:18.031092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.031081    5534 scope.go:122] "RemoveContainer" containerID="83493a7773c827cf47425e9abec1ab86cf6f54cd10e0b71737c039ac56770dc1"
Oct 30 18:00:18.031469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.031453    5534 scope.go:122] "RemoveContainer" containerID="d68fa9c2d891634aa6a62fa2c000f27d745c34d3f0be5a157684715876c0c537"
Oct 30 18:00:18.031851 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:18.031828    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs_odh-ai-gateway-infra(3f7532b0-bf21-420f-a014-d4c00efc644d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d"
Oct 30 18:00:18.072406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.072365    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn\") pod \"payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:18.230460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.230417    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:18.269359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.269335    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:18.285094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.285052    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:18.319448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.319415    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca\") pod \"6f1e8336-6765-4656-a1d3-28fead2c7d26\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") "
Oct 30 18:00:18.319628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.319525    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8\" (UniqueName: \"kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8\") pod \"6f1e8336-6765-4656-a1d3-28fead2c7d26\" (UID: \"6f1e8336-6765-4656-a1d3-28fead2c7d26\") "
Oct 30 18:00:18.319628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.319564    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca\") pod \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") "
Oct 30 18:00:18.319837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.319815    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m\" (UniqueName: \"kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m\") pod \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\" (UID: \"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e\") "
Oct 30 18:00:18.319934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.319913    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca" pod "6f1e8336-6765-4656-a1d3-28fead2c7d26" (UID: "6f1e8336-6765-4656-a1d3-28fead2c7d26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:18.320017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.319930    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca" pod "e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" (UID: "e9cf8f81-e7d4-40b1-850d-6cc1126cb60e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:18.320567 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.320545    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:18.320685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.320573    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e8336-6765-4656-a1d3-28fead2c7d26-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:18.322847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.322782    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8" pod "6f1e8336-6765-4656-a1d3-28fead2c7d26" (UID: "6f1e8336-6765-4656-a1d3-28fead2c7d26"). InnerVolumeSpecName "kube-api-access-z52j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:18.322847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.322826    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m" pod "e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" (UID: "e9cf8f81-e7d4-40b1-850d-6cc1126cb60e"). InnerVolumeSpecName "kube-api-access-7jg9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:18.339222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.339186    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:18.388887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.385605    5534 scope.go:122] "RemoveContainer" containerID="1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb"
Oct 30 18:00:18.427881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.427790    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z52j8\" (UniqueName: \"kubernetes.io/projected/6f1e8336-6765-4656-a1d3-28fead2c7d26-kube-api-access-z52j8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:18.427881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.427826    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg9m\" (UniqueName: \"kubernetes.io/projected/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e-kube-api-access-7jg9m\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:18.601757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:18.601727    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"]
Oct 30 18:00:18.662486 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:18.661552    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c09984f_edfd_4b00_b553_b328608173d2.slice/crio-7e5d9ce165ae9f9a47c662dab6047f368c6333b976e1d5a8a9ce7bb0d6d6104c WatchSource:0}: Error finding container 7e5d9ce165ae9f9a47c662dab6047f368c6333b976e1d5a8a9ce7bb0d6d6104c: Status 404 returned error can't find the container with id 7e5d9ce165ae9f9a47c662dab6047f368c6333b976e1d5a8a9ce7bb0d6d6104c
Oct 30 18:00:19.035557 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.035507    5534 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77285cce_053d_4ea7_85d7_f0d4b38e7aa1.slice/crio-conmon-89b5fa1b8f24ce7a0d4cf4da4ab35a20e40c142971b6809d7647236084e5fb0b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7783598_4314_49e8_9915_da38199cca33.slice/crio-conmon-a1d3dd94e6519dc5ee86124bafd31c89f189d6e3ecb0314622c89e21ee01ca2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3e5eed_bef2_4406_9baf_2d1008aafac9.slice/crio-conmon-7d0810d82659f821a501d1239ee599fa1f28460707055c269de183d47fb6c43d.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:00:19.044921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.044886    5534 generic.go:356] "Generic (PLEG): container finished" podID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerID="d0a825579b110248fb4cf40c0826169830763abf008af5b8cd217beb9e9db5d2" exitCode=60
Oct 30 18:00:19.045170 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.044917    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerDied","Data":"d0a825579b110248fb4cf40c0826169830763abf008af5b8cd217beb9e9db5d2"}
Oct 30 18:00:19.045170 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.045001    5534 scope.go:122] "RemoveContainer" containerID="3b2d3e9e67d83a19806008aa253cf304229f6ebb6482c0f4ca94eb0eb594ba12"
Oct 30 18:00:19.045457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.045430    5534 scope.go:122] "RemoveContainer" containerID="d0a825579b110248fb4cf40c0826169830763abf008af5b8cd217beb9e9db5d2"
Oct 30 18:00:19.045812 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.045788    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-wl9fr_odh-ai-gateway-infra(96cdda84-ee1f-43f0-83ef-b235c50d3f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85"
Oct 30 18:00:19.047884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.046834    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"
Oct 30 18:00:19.047884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.047385    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b" event={"ID":"e9cf8f81-e7d4-40b1-850d-6cc1126cb60e","Type":"ContainerDied","Data":"be848991466a226170fdf46cb35f7861d51f15c90af92ff92d8ea893042c0bfc"}
Oct 30 18:00:19.054924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.054898    5534 generic.go:356] "Generic (PLEG): container finished" podID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerID="59d37cdfa0236848901fd69fdc5d3bb6924acc6e633953d9150f153c867a3a3a" exitCode=60
Oct 30 18:00:19.055056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.055040    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerDied","Data":"59d37cdfa0236848901fd69fdc5d3bb6924acc6e633953d9150f153c867a3a3a"}
Oct 30 18:00:19.055407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.055389    5534 scope.go:122] "RemoveContainer" containerID="59d37cdfa0236848901fd69fdc5d3bb6924acc6e633953d9150f153c867a3a3a"
Oct 30 18:00:19.055700 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.055677    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m_odh-ai-gateway-infra(fbe4fbe3-73ec-4d07-b82a-6d290ffa4952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952"
Oct 30 18:00:19.058827 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.058800    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerStarted","Data":"0bde3022dcfd3152b19e51f368cfe3229fbef9fb59bf56d7c20c1622fb216af1"}
Oct 30 18:00:19.061218 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.061181    5534 generic.go:356] "Generic (PLEG): container finished" podID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerID="d02816cfc23bf87a39bea6dedea2b79ae49163b38945a4071e86cd65be497165" exitCode=60
Oct 30 18:00:19.061320 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.061248    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerDied","Data":"d02816cfc23bf87a39bea6dedea2b79ae49163b38945a4071e86cd65be497165"}
Oct 30 18:00:19.061616 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.061598    5534 scope.go:122] "RemoveContainer" containerID="d02816cfc23bf87a39bea6dedea2b79ae49163b38945a4071e86cd65be497165"
Oct 30 18:00:19.061911 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.061892    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm_odh-ai-gateway-infra(9ac0500a-7c7b-4f5e-94ec-8ae263349221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221"
Oct 30 18:00:19.063766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.063744    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"
Oct 30 18:00:19.064144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.064119    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg" event={"ID":"6f1e8336-6765-4656-a1d3-28fead2c7d26","Type":"ContainerDied","Data":"8ef42b24968a3b072c41901c799f1beb91c60343b004a9b7ec926dd65b7cd75d"}
Oct 30 18:00:19.066336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.066311    5534 generic.go:356] "Generic (PLEG): container finished" podID="d7783598-4314-49e8-9915-da38199cca33" containerID="a1d3dd94e6519dc5ee86124bafd31c89f189d6e3ecb0314622c89e21ee01ca2e" exitCode=60
Oct 30 18:00:19.066454 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.066381    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" event={"ID":"d7783598-4314-49e8-9915-da38199cca33","Type":"ContainerDied","Data":"a1d3dd94e6519dc5ee86124bafd31c89f189d6e3ecb0314622c89e21ee01ca2e"}
Oct 30 18:00:19.066799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.066781    5534 scope.go:122] "RemoveContainer" containerID="a1d3dd94e6519dc5ee86124bafd31c89f189d6e3ecb0314622c89e21ee01ca2e"
Oct 30 18:00:19.067071 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.067049    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2_odh-ai-gateway-infra(d7783598-4314-49e8-9915-da38199cca33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" podUID="d7783598-4314-49e8-9915-da38199cca33"
Oct 30 18:00:19.069493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.069470    5534 generic.go:356] "Generic (PLEG): container finished" podID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerID="89b5fa1b8f24ce7a0d4cf4da4ab35a20e40c142971b6809d7647236084e5fb0b" exitCode=60
Oct 30 18:00:19.069598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.069490    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" event={"ID":"77285cce-053d-4ea7-85d7-f0d4b38e7aa1","Type":"ContainerDied","Data":"89b5fa1b8f24ce7a0d4cf4da4ab35a20e40c142971b6809d7647236084e5fb0b"}
Oct 30 18:00:19.069913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.069893    5534 scope.go:122] "RemoveContainer" containerID="89b5fa1b8f24ce7a0d4cf4da4ab35a20e40c142971b6809d7647236084e5fb0b"
Oct 30 18:00:19.070318 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.070296    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7_odh-ai-gateway-infra(77285cce-053d-4ea7-85d7-f0d4b38e7aa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1"
Oct 30 18:00:19.070619 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.070601    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"]
Oct 30 18:00:19.071963 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.071943    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerID="7d0810d82659f821a501d1239ee599fa1f28460707055c269de183d47fb6c43d" exitCode=60
Oct 30 18:00:19.072051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.072018    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerDied","Data":"7d0810d82659f821a501d1239ee599fa1f28460707055c269de183d47fb6c43d"}
Oct 30 18:00:19.072412 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.072393    5534 scope.go:122] "RemoveContainer" containerID="7d0810d82659f821a501d1239ee599fa1f28460707055c269de183d47fb6c43d"
Oct 30 18:00:19.072886 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.072862    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr_odh-ai-gateway-infra(2f3e5eed-bef2-4406-9baf-2d1008aafac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9"
Oct 30 18:00:19.074429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.074407    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944" event={"ID":"9c09984f-edfd-4b00-b553-b328608173d2","Type":"ContainerStarted","Data":"6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a"}
Oct 30 18:00:19.074500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.074448    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944" event={"ID":"9c09984f-edfd-4b00-b553-b328608173d2","Type":"ContainerStarted","Data":"7e5d9ce165ae9f9a47c662dab6047f368c6333b976e1d5a8a9ce7bb0d6d6104c"}
Oct 30 18:00:19.074944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.074928    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:19.076957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.076937    5534 generic.go:356] "Generic (PLEG): container finished" podID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerID="a0ee46910a5c91c259d38d08b54223470bbff9ab643e92b0f296ed1f653d2053" exitCode=60
Oct 30 18:00:19.077070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.077006    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerDied","Data":"a0ee46910a5c91c259d38d08b54223470bbff9ab643e92b0f296ed1f653d2053"}
Oct 30 18:00:19.077343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.077325    5534 scope.go:122] "RemoveContainer" containerID="a0ee46910a5c91c259d38d08b54223470bbff9ab643e92b0f296ed1f653d2053"
Oct 30 18:00:19.077608 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:19.077582    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh_odh-ai-gateway-infra(6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a"
Oct 30 18:00:19.177019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.176963    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944" podStartSLOduration=2.176945127 podStartE2EDuration="2.176945127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:19.173546533 +0000 UTC m=+2824.935590357" watchObservedRunningTime="2026-10-30 18:00:19.176945127 +0000 UTC m=+2824.938988944"
Oct 30 18:00:19.189297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.189270    5534 scope.go:122] "RemoveContainer" containerID="846948f95c3e10517b46c75d6f8d7441db874b4f068d909c6de57b011dbf4448"
Oct 30 18:00:19.204437 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.204414    5534 scope.go:122] "RemoveContainer" containerID="3ed85f8e662d086256190e28cb2700a53754838822c3036aedac0c66f27e10ee"
Oct 30 18:00:19.206431 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.206407    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"]
Oct 30 18:00:19.223153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.223129    5534 scope.go:122] "RemoveContainer" containerID="c7a05ccb62c2be0efa29a124c732cce7af1206c88439e6d4e280b7dbce870586"
Oct 30 18:00:19.240056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.240032    5534 scope.go:122] "RemoveContainer" containerID="e6321ab510654fe29f2ebc679c672eab39da7e0bc4d8a478067eed1796bf9fc3"
Oct 30 18:00:19.252501 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.252466    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"]
Oct 30 18:00:19.256853 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.256144    5534 scope.go:122] "RemoveContainer" containerID="1a0719d0f1e73a31cda2a38a80bbcad5c2beb5c6389e55adbe37e887500eb52f"
Oct 30 18:00:19.277588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.276549    5534 scope.go:122] "RemoveContainer" containerID="84890e51fdb5b13901510428ffc30761d0a7a7caaedba6da4c47c90c038a73f7"
Oct 30 18:00:19.317719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.316709    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"]
Oct 30 18:00:19.325075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.325012    5534 scope.go:122] "RemoveContainer" containerID="42555b021369260e240f3d62df42685c01094efbddc8760698e85d22d33f92e3"
Oct 30 18:00:19.348944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.348919    5534 scope.go:122] "RemoveContainer" containerID="56c8a663f4abbd09d8d881660ac1e0d933e43e5d644e9b709441f2f691099e39"
Oct 30 18:00:19.370893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.370864    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:19.391062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.391033    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:19.443581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.443544    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca\") pod \"3f7532b0-bf21-420f-a014-d4c00efc644d\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") "
Oct 30 18:00:19.443873 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.443819    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9\" (UniqueName: \"kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9\") pod \"3f7532b0-bf21-420f-a014-d4c00efc644d\" (UID: \"3f7532b0-bf21-420f-a014-d4c00efc644d\") "
Oct 30 18:00:19.444320 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.444293    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca" pod "3f7532b0-bf21-420f-a014-d4c00efc644d" (UID: "3f7532b0-bf21-420f-a014-d4c00efc644d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:19.444690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.444612    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7532b0-bf21-420f-a014-d4c00efc644d-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:19.446507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.446485    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9" pod "3f7532b0-bf21-420f-a014-d4c00efc644d" (UID: "3f7532b0-bf21-420f-a014-d4c00efc644d"). InnerVolumeSpecName "kube-api-access-m7vr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:19.502081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.501950    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"]
Oct 30 18:00:19.509286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.509253    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0e7403-29889720-vw2vg"]
Oct 30 18:00:19.542722 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.542680    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"]
Oct 30 18:00:19.548508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.548469    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vr9\" (UniqueName: \"kubernetes.io/projected/3f7532b0-bf21-420f-a014-d4c00efc644d-kube-api-access-m7vr9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:19.549714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.549607    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c31c73-29889720-glk2b"]
Oct 30 18:00:19.550793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.550764    5534 scope.go:122] "RemoveContainer" containerID="f0a5bb4a1e65ecf5aaf5f86e2bd77085af520844df1f6d8e3dfaed9b85ba6671"
Oct 30 18:00:19.731961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.731933    5534 scope.go:122] "RemoveContainer" containerID="4e70a04b647f8e95f1e3037ca3ffde9033a13031dd5db1cc811c1b5b9b701bc0"
Oct 30 18:00:19.744995 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.744967    5534 scope.go:122] "RemoveContainer" containerID="4bcc742484f2f3ca99ac1b8b1316e45f584f332798b36c7bd7ad49ab5bdf0571"
Oct 30 18:00:19.908718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:19.908687    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:00:20.102910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.102872    5534 generic.go:356] "Generic (PLEG): container finished" podID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerID="0bde3022dcfd3152b19e51f368cfe3229fbef9fb59bf56d7c20c1622fb216af1" exitCode=60
Oct 30 18:00:20.103243 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.103221    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerDied","Data":"0bde3022dcfd3152b19e51f368cfe3229fbef9fb59bf56d7c20c1622fb216af1"}
Oct 30 18:00:20.103449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.103434    5534 scope.go:122] "RemoveContainer" containerID="1cf0675c0347d6107a9c1bf287982e868e6306fd4acdc014619164d07758d3cb"
Oct 30 18:00:20.106410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.106391    5534 scope.go:122] "RemoveContainer" containerID="0bde3022dcfd3152b19e51f368cfe3229fbef9fb59bf56d7c20c1622fb216af1"
Oct 30 18:00:20.107559 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:20.107531    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45_odh-ai-gateway-infra(d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3"
Oct 30 18:00:20.110572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.110547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:20.126289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.126259    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"
Oct 30 18:00:20.126937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.126904    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs" event={"ID":"3f7532b0-bf21-420f-a014-d4c00efc644d","Type":"ContainerDied","Data":"974ab774cdb9d71c427f90700897d0d0ae70877f80b3e0d14b714c8fa51cc0a8"}
Oct 30 18:00:20.127078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.126975    5534 scope.go:122] "RemoveContainer" containerID="d68fa9c2d891634aa6a62fa2c000f27d745c34d3f0be5a157684715876c0c537"
Oct 30 18:00:20.183001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.182968    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"]
Oct 30 18:00:20.233282 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.231956    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"]
Oct 30 18:00:20.280148 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.280067    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"]
Oct 30 18:00:20.292433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.292325    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-23c1e8-29889720-6vggs"]
Oct 30 18:00:20.317665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.317595    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"]
Oct 30 18:00:20.447891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.444352    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" path="/var/lib/kubelet/pods/3f7532b0-bf21-420f-a014-d4c00efc644d/volumes"
Oct 30 18:00:20.447891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.445303    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" path="/var/lib/kubelet/pods/6f1e8336-6765-4656-a1d3-28fead2c7d26/volumes"
Oct 30 18:00:20.447891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.445971    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" path="/var/lib/kubelet/pods/e9cf8f81-e7d4-40b1-850d-6cc1126cb60e/volumes"
Oct 30 18:00:20.447891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.446941    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"]
Oct 30 18:00:20.503772 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.502551    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:20.508579 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.508547    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"]
Oct 30 18:00:20.620358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.617455    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp\" (UniqueName: \"kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp\") pod \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") "
Oct 30 18:00:20.620358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.618017    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca\") pod \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\" (UID: \"9ac0500a-7c7b-4f5e-94ec-8ae263349221\") "
Oct 30 18:00:20.620358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.618390    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca" pod "9ac0500a-7c7b-4f5e-94ec-8ae263349221" (UID: "9ac0500a-7c7b-4f5e-94ec-8ae263349221"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.620358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.618948    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac0500a-7c7b-4f5e-94ec-8ae263349221-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.636417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.633780    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp" pod "9ac0500a-7c7b-4f5e-94ec-8ae263349221" (UID: "9ac0500a-7c7b-4f5e-94ec-8ae263349221"). InnerVolumeSpecName "kube-api-access-hv7jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:20.697823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.697439    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:20.717261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.717232    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:20.719740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.719683    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh\" (UniqueName: \"kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh\") pod \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") "
Oct 30 18:00:20.719925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.719851    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca\") pod \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\" (UID: \"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952\") "
Oct 30 18:00:20.720687 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.720612    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7jp\" (UniqueName: \"kubernetes.io/projected/9ac0500a-7c7b-4f5e-94ec-8ae263349221-kube-api-access-hv7jp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.720824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.720762    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca" pod "fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" (UID: "fbe4fbe3-73ec-4d07-b82a-6d290ffa4952"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.723290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.723179    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:20.723702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.723628    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh" pod "fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" (UID: "fbe4fbe3-73ec-4d07-b82a-6d290ffa4952"). InnerVolumeSpecName "kube-api-access-rgnqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:20.728968 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.728922    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:20.821718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.821683    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k\" (UniqueName: \"kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k\") pod \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") "
Oct 30 18:00:20.821906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.821816    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq\" (UniqueName: \"kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq\") pod \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") "
Oct 30 18:00:20.821984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.821960    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca\") pod \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\" (UID: \"96cdda84-ee1f-43f0-83ef-b235c50d3f85\") "
Oct 30 18:00:20.822699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.822670    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca" pod "96cdda84-ee1f-43f0-83ef-b235c50d3f85" (UID: "96cdda84-ee1f-43f0-83ef-b235c50d3f85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.822871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.822854    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca\") pod \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") "
Oct 30 18:00:20.822951 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.822917    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca\") pod \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\" (UID: \"77285cce-053d-4ea7-85d7-f0d4b38e7aa1\") "
Oct 30 18:00:20.823026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.822950    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854\" (UniqueName: \"kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854\") pod \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\" (UID: \"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a\") "
Oct 30 18:00:20.823175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823145    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca" pod "6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" (UID: "6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.823401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823380    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca" pod "77285cce-053d-4ea7-85d7-f0d4b38e7aa1" (UID: "77285cce-053d-4ea7-85d7-f0d4b38e7aa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.823841 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823820    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgnqh\" (UniqueName: \"kubernetes.io/projected/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-kube-api-access-rgnqh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.823937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823844    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cdda84-ee1f-43f0-83ef-b235c50d3f85-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.823937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823860    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.823937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823874    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.823937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.823887    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.826555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.826527    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854" pod "6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" (UID: "6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a"). InnerVolumeSpecName "kube-api-access-bd854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:20.828590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.828501    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq" pod "77285cce-053d-4ea7-85d7-f0d4b38e7aa1" (UID: "77285cce-053d-4ea7-85d7-f0d4b38e7aa1"). InnerVolumeSpecName "kube-api-access-htrcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:20.830151 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.830124    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k" pod "96cdda84-ee1f-43f0-83ef-b235c50d3f85" (UID: "96cdda84-ee1f-43f0-83ef-b235c50d3f85"). InnerVolumeSpecName "kube-api-access-pgl4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:20.830544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.830516    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:20.872782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.872741    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:20.925336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.925304    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca\") pod \"d7783598-4314-49e8-9915-da38199cca33\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") "
Oct 30 18:00:20.925568 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.925384    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp\" (UniqueName: \"kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp\") pod \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") "
Oct 30 18:00:20.925568 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.925493    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca\") pod \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\" (UID: \"2f3e5eed-bef2-4406-9baf-2d1008aafac9\") "
Oct 30 18:00:20.925736 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.925717    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8\" (UniqueName: \"kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8\") pod \"d7783598-4314-49e8-9915-da38199cca33\" (UID: \"d7783598-4314-49e8-9915-da38199cca33\") "
Oct 30 18:00:20.925800 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.925710    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca" pod "d7783598-4314-49e8-9915-da38199cca33" (UID: "d7783598-4314-49e8-9915-da38199cca33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.925908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.925886    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca" pod "2f3e5eed-bef2-4406-9baf-2d1008aafac9" (UID: "2f3e5eed-bef2-4406-9baf-2d1008aafac9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:20.926285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.926266    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgl4k\" (UniqueName: \"kubernetes.io/projected/96cdda84-ee1f-43f0-83ef-b235c50d3f85-kube-api-access-pgl4k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.926361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.926289    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3e5eed-bef2-4406-9baf-2d1008aafac9-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.926361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.926304    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrcq\" (UniqueName: \"kubernetes.io/projected/77285cce-053d-4ea7-85d7-f0d4b38e7aa1-kube-api-access-htrcq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.926361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.926318    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd854\" (UniqueName: \"kubernetes.io/projected/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a-kube-api-access-bd854\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.926361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.926332    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7783598-4314-49e8-9915-da38199cca33-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:20.928134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.928108    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8" pod "d7783598-4314-49e8-9915-da38199cca33" (UID: "d7783598-4314-49e8-9915-da38199cca33"). InnerVolumeSpecName "kube-api-access-wtsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:20.928517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:20.928474    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp" pod "2f3e5eed-bef2-4406-9baf-2d1008aafac9" (UID: "2f3e5eed-bef2-4406-9baf-2d1008aafac9"). InnerVolumeSpecName "kube-api-access-q6htp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:21.028203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.028105    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsx8\" (UniqueName: \"kubernetes.io/projected/d7783598-4314-49e8-9915-da38199cca33-kube-api-access-wtsx8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:21.028203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.028147    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6htp\" (UniqueName: \"kubernetes.io/projected/2f3e5eed-bef2-4406-9baf-2d1008aafac9-kube-api-access-q6htp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:21.149274 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.149247    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"
Oct 30 18:00:21.149411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.149250    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm" event={"ID":"9ac0500a-7c7b-4f5e-94ec-8ae263349221","Type":"ContainerDied","Data":"23fd2086849a808903338047dbdc32d0163e11f2775ba453b0fd6f5b78d17b28"}
Oct 30 18:00:21.149411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.149394    5534 scope.go:122] "RemoveContainer" containerID="d02816cfc23bf87a39bea6dedea2b79ae49163b38945a4071e86cd65be497165"
Oct 30 18:00:21.151008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.150971    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"
Oct 30 18:00:21.151128 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.151040    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2" event={"ID":"d7783598-4314-49e8-9915-da38199cca33","Type":"ContainerDied","Data":"3cfa459ebf4c4112271e467e25bc44d8e56740246f67bf3be1699cb64dc7e1fd"}
Oct 30 18:00:21.153722 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.153687    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7" event={"ID":"77285cce-053d-4ea7-85d7-f0d4b38e7aa1","Type":"ContainerDied","Data":"86a02c7b88605c56b11b6f523dcc5383f3e5adbc080a86d1e23e673c56c6bd58"}
Oct 30 18:00:21.153844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.153802    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"
Oct 30 18:00:21.159325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.159291    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr" event={"ID":"2f3e5eed-bef2-4406-9baf-2d1008aafac9","Type":"ContainerDied","Data":"a4ba52f7d26c33240d9443032007677277518bba7d800d29a864447d06eca872"}
Oct 30 18:00:21.159445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.159417    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"
Oct 30 18:00:21.166796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.166754    5534 scope.go:122] "RemoveContainer" containerID="a1d3dd94e6519dc5ee86124bafd31c89f189d6e3ecb0314622c89e21ee01ca2e"
Oct 30 18:00:21.172834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.171979    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh" event={"ID":"6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a","Type":"ContainerDied","Data":"ebb479d46c82f426c68b23c1ab01fc9a8bb5685807f483874a1bb81a7bcaae92"}
Oct 30 18:00:21.172834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.172117    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"
Oct 30 18:00:21.186668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.181588    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr" event={"ID":"96cdda84-ee1f-43f0-83ef-b235c50d3f85","Type":"ContainerDied","Data":"31bb51a01bca79c1d996bcb89a974579650c07f5bf4947f0e8f38b61b883f47a"}
Oct 30 18:00:21.186668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.181733    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"
Oct 30 18:00:21.190889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.190399    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"
Oct 30 18:00:21.195085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.194534    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m" event={"ID":"fbe4fbe3-73ec-4d07-b82a-6d290ffa4952","Type":"ContainerDied","Data":"d23ceca6b7414ca5a8032e592b0b1558aeb6cab77e17d4935a9b406366fa600f"}
Oct 30 18:00:21.205203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.205180    5534 scope.go:122] "RemoveContainer" containerID="89b5fa1b8f24ce7a0d4cf4da4ab35a20e40c142971b6809d7647236084e5fb0b"
Oct 30 18:00:21.219451 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.219429    5534 scope.go:122] "RemoveContainer" containerID="7d0810d82659f821a501d1239ee599fa1f28460707055c269de183d47fb6c43d"
Oct 30 18:00:21.232509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.232487    5534 scope.go:122] "RemoveContainer" containerID="a0ee46910a5c91c259d38d08b54223470bbff9ab643e92b0f296ed1f653d2053"
Oct 30 18:00:21.292446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.292419    5534 scope.go:122] "RemoveContainer" containerID="d0a825579b110248fb4cf40c0826169830763abf008af5b8cd217beb9e9db5d2"
Oct 30 18:00:21.295070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.295047    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"]
Oct 30 18:00:21.312712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.312475    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-455e4a-29889720-k7tpr"]
Oct 30 18:00:21.320083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.320050    5534 scope.go:122] "RemoveContainer" containerID="59d37cdfa0236848901fd69fdc5d3bb6924acc6e633953d9150f153c867a3a3a"
Oct 30 18:00:21.336890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.336853    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"]
Oct 30 18:00:21.343991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.343961    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016f1d-29889720-xpsq7"]
Oct 30 18:00:21.355241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.355204    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:21.389779 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.389748    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"]
Oct 30 18:00:21.396072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.396045    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a2a6cb-29889720-66kfm"]
Oct 30 18:00:21.425472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.425438    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"]
Oct 30 18:00:21.435659 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.435612    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27567a-29889720-lmkx2"]
Oct 30 18:00:21.437962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.437922    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca\") pod \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") "
Oct 30 18:00:21.438162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.438143    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9\" (UniqueName: \"kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9\") pod \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\" (UID: \"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3\") "
Oct 30 18:00:21.441464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.439114    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca" pod "d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" (UID: "d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:00:21.444334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.444296    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9" pod "d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" (UID: "d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3"). InnerVolumeSpecName "kube-api-access-ntfv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:21.475525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.475493    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"]
Oct 30 18:00:21.482586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.482548    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ce0480-29889720-qmsmh"]
Oct 30 18:00:21.541526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.541435    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:21.541526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.541481    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfv9\" (UniqueName: \"kubernetes.io/projected/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3-kube-api-access-ntfv9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:21.547682 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.547613    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"]
Oct 30 18:00:21.554145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.554114    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-wl9fr"]
Oct 30 18:00:21.591229 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.591182    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"]
Oct 30 18:00:21.596545 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.596506    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-4dcdbb-29889720-2mf2m"]
Oct 30 18:00:21.908729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:21.908697    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:22.201337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.201243    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45" event={"ID":"d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3","Type":"ContainerDied","Data":"90205a5d590c0f45c637a260c4ea86a81fc917be6470dc8dade85a6c06ca5299"}
Oct 30 18:00:22.201337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.201305    5534 scope.go:122] "RemoveContainer" containerID="0bde3022dcfd3152b19e51f368cfe3229fbef9fb59bf56d7c20c1622fb216af1"
Oct 30 18:00:22.201337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.201255    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"
Oct 30 18:00:22.254591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.254553    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"]
Oct 30 18:00:22.265056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.265022    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889720-bzt45"]
Oct 30 18:00:22.375371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.375341    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" path="/var/lib/kubelet/pods/2f3e5eed-bef2-4406-9baf-2d1008aafac9/volumes"
Oct 30 18:00:22.375888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.375871    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" path="/var/lib/kubelet/pods/6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a/volumes"
Oct 30 18:00:22.376296 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.376283    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" path="/var/lib/kubelet/pods/77285cce-053d-4ea7-85d7-f0d4b38e7aa1/volumes"
Oct 30 18:00:22.377110 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.377091    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" path="/var/lib/kubelet/pods/96cdda84-ee1f-43f0-83ef-b235c50d3f85/volumes"
Oct 30 18:00:22.377470 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.377458    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" path="/var/lib/kubelet/pods/9ac0500a-7c7b-4f5e-94ec-8ae263349221/volumes"
Oct 30 18:00:22.377894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.377881    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" path="/var/lib/kubelet/pods/d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3/volumes"
Oct 30 18:00:22.378523 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.378509    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7783598-4314-49e8-9915-da38199cca33" path="/var/lib/kubelet/pods/d7783598-4314-49e8-9915-da38199cca33/volumes"
Oct 30 18:00:22.378916 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:22.378904    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" path="/var/lib/kubelet/pods/fbe4fbe3-73ec-4d07-b82a-6d290ffa4952/volumes"
Oct 30 18:00:23.627474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:23.627444    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:24.441862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:24.441828    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:24.882664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:24.882547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:24.999211 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:24.999169    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:25.175818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:25.168465    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:25.583188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:25.583090    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:27.127157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:27.127127    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:27.852730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:27.852666    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:28.868130 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:28.868078    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:29.003208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:29.003176    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:29.798746 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:29.798696    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:30.632666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.632423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:30.760615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.760563    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:30.962724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.961977    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"]
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962904    5534 cpu_manager.go:414] "containerMap: removing container" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962926    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962939    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962945    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962954    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962959    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962968    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962972    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962984    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.962991    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963000    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963004    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963011    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963016    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963025    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963029    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963040    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963045    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963056    5534 cpu_manager.go:414] "containerMap: removing container" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963063    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963077    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963082    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963090    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963094    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963099    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963103    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963113    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963118    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963129    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963133    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963145    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963151    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963164    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963169    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963177    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963181    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963188    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963192    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963198    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963203    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963210    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963216    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963232    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963236    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963242    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963247    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963255    5534 cpu_manager.go:414] "containerMap: removing container" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963259    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963268    5534 cpu_manager.go:414] "containerMap: removing container" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963272    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963399    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963409    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963417    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963422    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963429    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963435    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963440    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963449    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963459    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963466    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963471    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963478    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963484    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963490    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963495    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963500    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34f34ba-1b9a-40d8-92d0-4b19f36eb6b3" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963505    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963510    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963516    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963520    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963529    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963536    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963542    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963547    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963552    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963558    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963565    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963571    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe4fbe3-73ec-4d07-b82a-6d290ffa4952" containerName="cleanup"
Oct 30 18:00:30.973684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.963579    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:31.037891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:30.979734    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.038964 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.038941    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5cbb4\""
Oct 30 18:00:31.042878 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.039172    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b5cbb4-dockercfg-mzz4l\""
Oct 30 18:00:31.070403 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.070366    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"]
Oct 30 18:00:31.081665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.081258    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntch\" (UniqueName: \"kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch\") pod \"payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.081665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.081337    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.183601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.183564    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntch\" (UniqueName: \"kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch\") pod \"payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.183825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.183668    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.184808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.184779    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.194395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.194368    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:00:31.207223 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.207191    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntch\" (UniqueName: \"kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch\") pod \"payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.386007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.385965    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:31.709401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.709364    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"]
Oct 30 18:00:31.710919 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:31.710883    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7db7158_d21c_4e36_af1d_27aa6b4dfe8c.slice/crio-e5a872841efeb72753d2ebadc91ae2f9dbb3dfe1513d12a5f3c1fc25b1d9cb01 WatchSource:0}: Error finding container e5a872841efeb72753d2ebadc91ae2f9dbb3dfe1513d12a5f3c1fc25b1d9cb01: Status 404 returned error can't find the container with id e5a872841efeb72753d2ebadc91ae2f9dbb3dfe1513d12a5f3c1fc25b1d9cb01
Oct 30 18:00:31.866307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:31.866273    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:32.364673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:32.361104    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v" event={"ID":"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c","Type":"ContainerStarted","Data":"54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c"}
Oct 30 18:00:32.364673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:32.361193    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v" event={"ID":"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c","Type":"ContainerStarted","Data":"e5a872841efeb72753d2ebadc91ae2f9dbb3dfe1513d12a5f3c1fc25b1d9cb01"}
Oct 30 18:00:32.382782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:32.382666    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:32.414736 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:32.414670    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v" podStartSLOduration=2.414620817 podStartE2EDuration="2.414620817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:32.413053826 +0000 UTC m=+2838.175097640" watchObservedRunningTime="2026-10-30 18:00:32.414620817 +0000 UTC m=+2838.176664632"
Oct 30 18:00:33.385879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:33.385825    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:33.562668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:33.560905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:34.103667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.103275    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"]
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.104917    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.104942    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105003    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105010    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1e8336-6765-4656-a1d3-28fead2c7d26" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105073    5534 cpu_manager.go:414] "containerMap: removing container" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105080    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105112    5534 cpu_manager.go:414] "containerMap: removing container" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105125    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cdda84-ee1f-43f0-83ef-b235c50d3f85" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105143    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105150    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac0500a-7c7b-4f5e-94ec-8ae263349221" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105174    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105181    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7783598-4314-49e8-9915-da38199cca33" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105209    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105217    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3e5eed-bef2-4406-9baf-2d1008aafac9" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105239    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105247    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105794    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cf8f81-e7d4-40b1-850d-6cc1126cb60e" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105820    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3ee434-6df1-4e9f-bfb3-8fddb68d5f2a" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105852    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7532b0-bf21-420f-a014-d4c00efc644d" containerName="cleanup"
Oct 30 18:00:34.106705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.105874    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="77285cce-053d-4ea7-85d7-f0d4b38e7aa1" containerName="cleanup"
Oct 30 18:00:34.122574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.122537    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.136278 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.136248    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"]
Oct 30 18:00:34.149331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.149272    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fab25f-dockercfg-h8xcz\""
Oct 30 18:00:34.149492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.149335    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fab25f\""
Oct 30 18:00:34.185766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.185721    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.193148 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.193108    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p768s\" (UniqueName: \"kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.294937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.294875    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.295173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.295060    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p768s\" (UniqueName: \"kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.296144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.296116    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.318503 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.318472    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p768s\" (UniqueName: \"kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.449345 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.449241    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:34.846422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.846386    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"]
Oct 30 18:00:34.846593 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:00:34.846539    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69598f5e_08bd_4fac_b74b_c9c30d5867e4.slice/crio-21b33d095836031b022228e151e42b5fffaa8e2b820970f50649b92cd1f19f3f WatchSource:0}: Error finding container 21b33d095836031b022228e151e42b5fffaa8e2b820970f50649b92cd1f19f3f: Status 404 returned error can't find the container with id 21b33d095836031b022228e151e42b5fffaa8e2b820970f50649b92cd1f19f3f
Oct 30 18:00:34.848627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:34.848598    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:35.414532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:35.414498    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" event={"ID":"69598f5e-08bd-4fac-b74b-c9c30d5867e4","Type":"ContainerStarted","Data":"c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938"}
Oct 30 18:00:35.414778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:35.414559    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" event={"ID":"69598f5e-08bd-4fac-b74b-c9c30d5867e4","Type":"ContainerStarted","Data":"21b33d095836031b022228e151e42b5fffaa8e2b820970f50649b92cd1f19f3f"}
Oct 30 18:00:35.414778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:35.414580    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:35.468670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:35.461914    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" podStartSLOduration=1.46189376 podStartE2EDuration="1.46189376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:35.460135053 +0000 UTC m=+2841.222178869" watchObservedRunningTime="2026-10-30 18:00:35.46189376 +0000 UTC m=+2841.223937576"
Oct 30 18:00:36.840008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:36.839977    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:38.196669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:38.195424    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"]
Oct 30 18:00:38.214028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:38.213976    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:38.924273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:38.924088    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:40.026049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:40.026012    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:40.635609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:40.635570    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:42.206338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:42.206300    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:44.066657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.066542    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:44.404723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.404683    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:00:44.515889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.515628    5534 generic.go:356] "Generic (PLEG): container finished" podID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerID="c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e" exitCode=0
Oct 30 18:00:44.515889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.515743    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" event={"ID":"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a","Type":"ContainerDied","Data":"c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"}
Oct 30 18:00:44.516334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.516314    5534 scope.go:122] "RemoveContainer" containerID="c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"
Oct 30 18:00:44.518525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.518492    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:00:44.518701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.518569    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:00:44.518701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:44.518629    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:00:45.246404 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:45.246367    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:00:45.528635 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:45.528547    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" event={"ID":"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a","Type":"ContainerStarted","Data":"9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada"}
Oct 30 18:00:45.945825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:45.945791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:46.206821 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:46.206709    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 18:00:46.207027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:46.206941    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 18:00:46.207027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:46.206998    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 18:00:46.424920 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:46.424872    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" podUID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.1.60:9004: connect: connection refused"
Oct 30 18:00:47.206248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:47.206210    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 18:00:47.206493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:47.206287    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 18:00:47.835925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:47.835893    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:48.207167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:48.207140    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 18:00:48.207395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:48.207374    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 18:00:48.544836 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:48.544747    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:48.587251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:48.587198    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" podUID="cd29d125f430f9e0a8a7b7b6580a8dec" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 18:00:49.032350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:49.032316    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"]
Oct 30 18:00:49.044270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:49.044240    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a204bd7-rk2vv"]
Oct 30 18:00:49.206186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:49.206143    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 18:00:49.206412 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:49.206202    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 18:00:49.801469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:49.801439    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:50.206487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:50.206453    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 18:00:50.207038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:50.206510    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 18:00:50.394222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:50.394189    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631a6569-80ae-41bc-991f-11d7b61c4cb2" path="/var/lib/kubelet/pods/631a6569-80ae-41bc-991f-11d7b61c4cb2/volumes"
Oct 30 18:00:51.213047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:51.213014    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 18:00:51.213584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:51.213566    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 18:00:51.214287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:51.214267    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 18:00:51.652496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:51.652463    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:53.231581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.231513    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5" podUID="1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" containerName="main" containerID="cri-o://826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d" gracePeriod=45
Oct 30 18:00:53.321323 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.321290    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:53.536887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.536858    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 18:00:53.559003 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.558968    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6\" (UniqueName: \"kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6\") pod \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") "
Oct 30 18:00:53.559217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559060    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache\") pod \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") "
Oct 30 18:00:53.559302 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559250    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs\") pod \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") "
Oct 30 18:00:53.559364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559321    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm\") pod \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") "
Oct 30 18:00:53.559364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559346    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home\") pod \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") "
Oct 30 18:00:53.559481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559419    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir\") pod \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\" (UID: \"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409\") "
Oct 30 18:00:53.559613 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559342    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache" pod "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" (UID: "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:53.559911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559886    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir" pod "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" (UID: "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:53.560207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.559920    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home" pod "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" (UID: "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:53.560713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.560690    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:53.560819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.560719    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:53.560819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.560733    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:53.561831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.561806    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6" pod "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" (UID: "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409"). InnerVolumeSpecName "kube-api-access-dqdv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:00:53.561930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.561829    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm" pod "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" (UID: "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:00:53.561979 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.561962    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs" pod "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" (UID: "1aa60a85-3c0b-4bfc-a756-f6f0cd83d409"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:00:53.633505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.633469    5534 generic.go:356] "Generic (PLEG): container finished" podID="1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" containerID="826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d" exitCode=0
Oct 30 18:00:53.633777 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.633534    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"
Oct 30 18:00:53.633852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.633772    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5" event={"ID":"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409","Type":"ContainerDied","Data":"826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d"}
Oct 30 18:00:53.633852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.633806    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5" event={"ID":"1aa60a85-3c0b-4bfc-a756-f6f0cd83d409","Type":"ContainerDied","Data":"8f83219461e2af03e55fc341e5352e685b94b652a9f5da2a499cffcf9aa1159a"}
Oct 30 18:00:53.633852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.633823    5534 scope.go:122] "RemoveContainer" containerID="826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d"
Oct 30 18:00:53.651125 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.650941    5534 scope.go:122] "RemoveContainer" containerID="826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d"
Oct 30 18:00:53.651418 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:00:53.651369    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d\": container with ID starting with 826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d not found: ID does not exist" containerID="826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d"
Oct 30 18:00:53.651563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.651414    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d"} err="failed to get container status \"826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d\": rpc error: code = NotFound desc = could not find container \"826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d\": container with ID starting with 826763c1a8e38cf66f6e92e243dc8e0749303b32090bbf72ecff6e9d1bba057d not found: ID does not exist"
Oct 30 18:00:53.661902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.661863    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdv6\" (UniqueName: \"kubernetes.io/projected/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-kube-api-access-dqdv6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:53.661902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.661895    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:53.661902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.661912    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:00:53.687143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.687106    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"]
Oct 30 18:00:53.691534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.691498    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/sub-test-model-23c1e8-kserve-58c749c88f-8dwb5"]
Oct 30 18:00:53.793436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:53.793347    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:54.375178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:54.375140    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" path="/var/lib/kubelet/pods/1aa60a85-3c0b-4bfc-a756-f6f0cd83d409/volumes"
Oct 30 18:00:55.751492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:55.751460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:56.423466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:56.423437    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:00:57.491938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:57.491904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:00:59.030430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:59.030391    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"]
Oct 30 18:00:59.041754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:59.041721    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3719eb9-k8bcd"]
Oct 30 18:00:59.115947 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:00:59.115911    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:00.373290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:00.373250    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10f806f-d450-4cd1-ab0c-3f8ef9713b49" path="/var/lib/kubelet/pods/a10f806f-d450-4cd1-ab0c-3f8ef9713b49/volumes"
Oct 30 18:01:00.894791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:00.894741    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:02.475701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:02.475665    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:03.117423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.117390    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"]
Oct 30 18:01:03.119156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.119128    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" containerName="main"
Oct 30 18:01:03.119156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.119154    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" containerName="main"
Oct 30 18:01:03.119368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.119357    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa60a85-3c0b-4bfc-a756-f6f0cd83d409" containerName="main"
Oct 30 18:01:03.124422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.124398    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.140085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.140058    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-23c1e8-openshift-default-dockercfg-4mhhv\""
Oct 30 18:01:03.157495 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.157458    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"]
Oct 30 18:01:03.220492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.220449    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.220755 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.220514    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.220755 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.220625    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndkr\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.220755 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.220729    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.220958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.220837    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.221013 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.220978    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.221048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.221020    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.221092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.221068    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.221155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.221138    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322108    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322165    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322213    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndkr\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322237    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322302    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322410    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322461    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322530    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322565    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322605    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.322901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322870    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.323024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322944    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.323024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.322948    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.323278 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.323255    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.325101 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.325072    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.325223 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.325195    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.345763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.345721    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.346077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.346039    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndkr\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.440191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.440077    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:03.777680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.777637    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"]
Oct 30 18:01:03.779302 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:01:03.779279    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1e2f63_08a6_44b7_9b08_ceb3f8d412b3.slice/crio-3db722ba77e9871399f495018e75015dc7f329ef22cebde963ba715a1ead8c7e WatchSource:0}: Error finding container 3db722ba77e9871399f495018e75015dc7f329ef22cebde963ba715a1ead8c7e: Status 404 returned error can't find the container with id 3db722ba77e9871399f495018e75015dc7f329ef22cebde963ba715a1ead8c7e
Oct 30 18:01:03.783244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.783207    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:03.783381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.783299    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:03.783381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:03.783356    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:04.178666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:04.177480    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:04.741167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:04.741123    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" event={"ID":"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3","Type":"ContainerStarted","Data":"ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898"}
Oct 30 18:01:04.741379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:04.741189    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" event={"ID":"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3","Type":"ContainerStarted","Data":"3db722ba77e9871399f495018e75015dc7f329ef22cebde963ba715a1ead8c7e"}
Oct 30 18:01:04.802164 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:04.802109    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podStartSLOduration=1.802093899 podStartE2EDuration="1.802093899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:04.797450023 +0000 UTC m=+2870.559493858" watchObservedRunningTime="2026-10-30 18:01:04.802093899 +0000 UTC m=+2870.564137705"
Oct 30 18:01:05.441017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:05.440981    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:05.442782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:05.442757    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 18:01:05.442931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:05.442819    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 18:01:06.107719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.107686    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:06.240814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.240771    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:06.360727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.359144    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:06.441408 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.441372    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 18:01:06.441600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.441437    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 18:01:06.603136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.603102    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"]
Oct 30 18:01:06.635376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.635286    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"]
Oct 30 18:01:06.701922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.700549    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:06.922512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:06.922421    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:07.444668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:07.444042    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 18:01:07.444668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:07.444101    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 18:01:08.441347 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:08.441305    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 18:01:08.441524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:08.441372    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 18:01:08.499665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:08.498772    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:08.832312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:08.832221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:08.946396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:08.946359    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:09.441315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:09.441281    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 18:01:09.441515 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:09.441333    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 18:01:09.999213 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:09.999176    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:10.253729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:10.248211    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:10.373666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:10.372727    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:10.440392 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:10.440359    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 18:01:10.440608 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:10.440423    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 18:01:10.757685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:10.756699    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:11.040345 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.040119    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:11.447959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.447927    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:11.448486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.448460    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:11.449442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.449421    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:01:11.607664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.606721    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7" podUID="6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" containerName="payload-processing" containerID="cri-o://3f26720898fe478046d17d29342b817b663ff9ad8844116fb173e959d66ee6ad" gracePeriod=25
Oct 30 18:01:11.636498 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.636462    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548" podUID="944f5a3a-2c51-4f25-80b6-665442097124" containerName="payload-pre-processing" containerID="cri-o://6f10493a431c78aa8799083aa58fff035d82593edb22b28ae264a4dfacb925f5" gracePeriod=25
Oct 30 18:01:11.948811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.948318    5534 generic.go:356] "Generic (PLEG): container finished" podID="944f5a3a-2c51-4f25-80b6-665442097124" containerID="6f10493a431c78aa8799083aa58fff035d82593edb22b28ae264a4dfacb925f5" exitCode=0
Oct 30 18:01:11.948811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.948627    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548" event={"ID":"944f5a3a-2c51-4f25-80b6-665442097124","Type":"ContainerDied","Data":"6f10493a431c78aa8799083aa58fff035d82593edb22b28ae264a4dfacb925f5"}
Oct 30 18:01:11.962671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.960054    5534 generic.go:356] "Generic (PLEG): container finished" podID="6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" containerID="3f26720898fe478046d17d29342b817b663ff9ad8844116fb173e959d66ee6ad" exitCode=0
Oct 30 18:01:11.962671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:11.961133    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7" event={"ID":"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3","Type":"ContainerDied","Data":"3f26720898fe478046d17d29342b817b663ff9ad8844116fb173e959d66ee6ad"}
Oct 30 18:01:12.011411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.011381    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 18:01:12.015606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.015574    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 18:01:12.103980 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.103948    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume\") pod \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") "
Oct 30 18:01:12.104493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.104056    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn\" (UniqueName: \"kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn\") pod \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\" (UID: \"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3\") "
Oct 30 18:01:12.104493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.104142    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume\") pod \"944f5a3a-2c51-4f25-80b6-665442097124\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") "
Oct 30 18:01:12.104493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.104415    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc\" (UniqueName: \"kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc\") pod \"944f5a3a-2c51-4f25-80b6-665442097124\" (UID: \"944f5a3a-2c51-4f25-80b6-665442097124\") "
Oct 30 18:01:12.104705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.104593    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume" pod "944f5a3a-2c51-4f25-80b6-665442097124" (UID: "944f5a3a-2c51-4f25-80b6-665442097124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:01:12.105507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.105476    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f5a3a-2c51-4f25-80b6-665442097124-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:12.106290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.106257    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume" pod "6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" (UID: "6d6712e2-5875-4cf4-ab9b-a5a88469a2b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:01:12.107747 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.107719    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn" pod "6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" (UID: "6d6712e2-5875-4cf4-ab9b-a5a88469a2b3"). InnerVolumeSpecName "kube-api-access-df9fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:01:12.109511 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.109477    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc" pod "944f5a3a-2c51-4f25-80b6-665442097124" (UID: "944f5a3a-2c51-4f25-80b6-665442097124"). InnerVolumeSpecName "kube-api-access-fn4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:01:12.135255 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.135219    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:12.214401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.214257    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4vc\" (UniqueName: \"kubernetes.io/projected/944f5a3a-2c51-4f25-80b6-665442097124-kube-api-access-fn4vc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:12.214401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.214298    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:12.214401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.214313    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9fn\" (UniqueName: \"kubernetes.io/projected/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3-kube-api-access-df9fn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:12.973982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.973948    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"
Oct 30 18:01:12.973982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.973979    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7" event={"ID":"6d6712e2-5875-4cf4-ab9b-a5a88469a2b3","Type":"ContainerDied","Data":"61600b2fb81a55464df73f92abb01e3db6723bc9b09bdf5ebd3e65acbcc49835"}
Oct 30 18:01:12.974276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.974040    5534 scope.go:122] "RemoveContainer" containerID="3f26720898fe478046d17d29342b817b663ff9ad8844116fb173e959d66ee6ad"
Oct 30 18:01:12.978816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.978774    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"
Oct 30 18:01:12.978816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.978791    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548" event={"ID":"944f5a3a-2c51-4f25-80b6-665442097124","Type":"ContainerDied","Data":"620c0f73e43e97e3eaf1a884861d325b085538d19120b1e8dd4d98a2535f6e92"}
Oct 30 18:01:12.990436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:12.990407    5534 scope.go:122] "RemoveContainer" containerID="6f10493a431c78aa8799083aa58fff035d82593edb22b28ae264a4dfacb925f5"
Oct 30 18:01:13.029381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:13.029345    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"]
Oct 30 18:01:13.039696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:13.039669    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-8486cfdb79-l5548"]
Oct 30 18:01:13.069550 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:13.069515    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"]
Oct 30 18:01:13.076266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:13.076232    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-65fdff966-fwtn7"]
Oct 30 18:01:13.911748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:13.911720    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:14.375090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:14.375060    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" path="/var/lib/kubelet/pods/6d6712e2-5875-4cf4-ab9b-a5a88469a2b3/volumes"
Oct 30 18:01:14.375489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:14.375475    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944f5a3a-2c51-4f25-80b6-665442097124" path="/var/lib/kubelet/pods/944f5a3a-2c51-4f25-80b6-665442097124/volumes"
Oct 30 18:01:14.516174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:14.516146    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:14.750035 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:14.749950    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:15.245864 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:01:15.245829    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:01:15.918533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:15.918507    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:16.236133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.236050    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:16.340180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.340147    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"]
Oct 30 18:01:16.340985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.340968    5534 cpu_manager.go:414] "containerMap: removing container" podUID="944f5a3a-2c51-4f25-80b6-665442097124" containerName="payload-pre-processing"
Oct 30 18:01:16.340985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.340985    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f5a3a-2c51-4f25-80b6-665442097124" containerName="payload-pre-processing"
Oct 30 18:01:16.341111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.341018    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" containerName="payload-processing"
Oct 30 18:01:16.341111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.341024    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" containerName="payload-processing"
Oct 30 18:01:16.341184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.341135    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="944f5a3a-2c51-4f25-80b6-665442097124" containerName="payload-pre-processing"
Oct 30 18:01:16.341184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.341149    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6712e2-5875-4cf4-ab9b-a5a88469a2b3" containerName="payload-processing"
Oct 30 18:01:16.346163 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.346147    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.353233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.353204    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:16.371765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.371735    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f10dda\""
Oct 30 18:01:16.377763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.377732    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"]
Oct 30 18:01:16.386310 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.386274    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7z6\" (UniqueName: \"kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6\") pod \"payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.386490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.386335    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.405591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.405559    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"]
Oct 30 18:01:16.410505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.410481    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.431276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.431245    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f10dda-dockercfg-tgbb9\""
Oct 30 18:01:16.436622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.436595    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"]
Oct 30 18:01:16.573726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.572955    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxpq\" (UniqueName: \"kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.573726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.573177    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7z6\" (UniqueName: \"kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6\") pod \"payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.573726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.573232    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.573726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.573378    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.598666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.575237    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.682718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.669774    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7z6\" (UniqueName: \"kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6\") pod \"payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:16.694692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.684792    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.694692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.685084    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxpq\" (UniqueName: \"kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.694692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.686603    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.735321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.735265    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxpq\" (UniqueName: \"kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:16.968517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:16.967805    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:17.035399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:17.034602    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:17.291154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:17.291120    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"]
Oct 30 18:01:17.292059 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:01:17.292028    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee5267d_e499_4cad_8e7f_ab4080952027.slice/crio-a84389f9c475e82f4e4048cbbe60e7ad14592a830e2d2889726a21d62b1dfc8c WatchSource:0}: Error finding container a84389f9c475e82f4e4048cbbe60e7ad14592a830e2d2889726a21d62b1dfc8c: Status 404 returned error can't find the container with id a84389f9c475e82f4e4048cbbe60e7ad14592a830e2d2889726a21d62b1dfc8c
Oct 30 18:01:17.387617 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:17.385564    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"]
Oct 30 18:01:18.026483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.026440    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:18.093399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.093359    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg" event={"ID":"cee5267d-e499-4cad-8e7f-ab4080952027","Type":"ContainerStarted","Data":"11fca12c12e04c8c5f708cc6e6f2886c849d6d50f7cad46064896dd49fe5d283"}
Oct 30 18:01:18.093634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.093429    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg" event={"ID":"cee5267d-e499-4cad-8e7f-ab4080952027","Type":"ContainerStarted","Data":"a84389f9c475e82f4e4048cbbe60e7ad14592a830e2d2889726a21d62b1dfc8c"}
Oct 30 18:01:18.093634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.093460    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:18.095260 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.095232    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns" event={"ID":"741e5fb2-53df-4dd8-8528-1f1f0c10afe9","Type":"ContainerStarted","Data":"91a82b89660f44ac0338402617445ed359b1609b7d8a047168ea92d05c012b20"}
Oct 30 18:01:18.095401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.095295    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:18.095401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.095318    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns" event={"ID":"741e5fb2-53df-4dd8-8528-1f1f0c10afe9","Type":"ContainerStarted","Data":"7db7b0da35b6b1301974b1d24679a84e4a024fdb816ebce479d9de0320e0e127"}
Oct 30 18:01:18.138977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.137084    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg" podStartSLOduration=2.137064231 podStartE2EDuration="2.137064231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:18.13291438 +0000 UTC m=+2883.894958194" watchObservedRunningTime="2026-10-30 18:01:18.137064231 +0000 UTC m=+2883.899108064"
Oct 30 18:01:18.187996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:18.187931    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns" podStartSLOduration=2.187910177 podStartE2EDuration="2.187910177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:18.18211431 +0000 UTC m=+2883.944158128" watchObservedRunningTime="2026-10-30 18:01:18.187910177 +0000 UTC m=+2883.949953991"
Oct 30 18:01:19.832114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:19.832080    5534 scope.go:122] "RemoveContainer" containerID="a3504f9ede1f042f8e21cbc799c55f836f6a79c3480a6dda11b73cb755803adb"
Oct 30 18:01:20.163045 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.163009    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"]
Oct 30 18:01:20.172600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.172573    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.193427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.192827    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b5cbb4-serving-cert\""
Oct 30 18:01:20.201698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.201611    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"]
Oct 30 18:01:20.245126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.245083    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.245339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.245153    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rfl\" (UniqueName: \"kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.245438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.245370    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.245528 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.245508    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.259706 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.259632    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:20.347253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.347219    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.347450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.347283    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rfl\" (UniqueName: \"kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.347450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.347440    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.347576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.347558    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.348573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.348537    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.350724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.350696    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.351496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.351468    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.370803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.370770    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rfl\" (UniqueName: \"kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl\") pod \"maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.394016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.383377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:20.517771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.517678    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:20.961063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:20.941572    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"]
Oct 30 18:01:21.318791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:21.318676    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" event={"ID":"94d38a2b-3e96-471a-aed1-c55be0699e11","Type":"ContainerStarted","Data":"cda51d0ffbc59a7fa1b6570c2586f63044547d89431af8c00274b305d1ab4ce8"}
Oct 30 18:01:22.331287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:22.331179    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" event={"ID":"94d38a2b-3e96-471a-aed1-c55be0699e11","Type":"ContainerStarted","Data":"0ba94c0d295eb1e53042bf332c5c83a103b3f1a8691eb4f8db17b90418a47fba"}
Oct 30 18:01:22.331897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:22.331284    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:22.374118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:22.374054    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" podStartSLOduration=1.455350299 podStartE2EDuration="2.374034547s" totalImagesPullingTime="918.684248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:22.372501467 +0000 UTC m=+2888.134545281" watchObservedRunningTime="2026-10-30 18:01:22.374034547 +0000 UTC m=+2888.136078364"
Oct 30 18:01:22.616976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:22.616949    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:24.271503 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:24.271466    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:25.942681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:25.942614    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:27.282306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:27.282274    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:28.314176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:28.314143    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:28.345358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:28.345331    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:01:29.043863 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.043826    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"]
Oct 30 18:01:29.049426 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.049403    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.068727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.068699    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"]
Oct 30 18:01:29.075143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.075117    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:01:29.075322 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.075242    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:01:29.109896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.109862    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:01:29.110405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.110381    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:01:29.122742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.122707    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.122930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.122818    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.123007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.122978    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjpd\" (UniqueName: \"kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.224404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.224363    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.224704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.224483    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.224815 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.224787    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjpd\" (UniqueName: \"kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.225131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.225105    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.234066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.234036    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.248153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.248127    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjpd\" (UniqueName: \"kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd\") pod \"maas-api-revoke-sub-9879d034-7wmdd\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.364998 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.364966    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:29.679235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.679209    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"]
Oct 30 18:01:29.680794 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:01:29.680767    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0670fb6b_0681_455e_a46e_b9a18805a1d5.slice/crio-4fb9ddeebf7fdc48275aa127beae95960332cc7b0a412c68ec9c983dea3dd80f WatchSource:0}: Error finding container 4fb9ddeebf7fdc48275aa127beae95960332cc7b0a412c68ec9c983dea3dd80f: Status 404 returned error can't find the container with id 4fb9ddeebf7fdc48275aa127beae95960332cc7b0a412c68ec9c983dea3dd80f
Oct 30 18:01:29.938474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:29.938389    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:30.051108 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:30.051079    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:30.454406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:30.454369    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd" event={"ID":"0670fb6b-0681-455e-a46e-b9a18805a1d5","Type":"ContainerStarted","Data":"318ae302e6fef49e500318d69fb324ccc14eeba4e9fa8fa6e97f3515d84b059d"}
Oct 30 18:01:30.454849 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:30.454432    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd" event={"ID":"0670fb6b-0681-455e-a46e-b9a18805a1d5","Type":"ContainerStarted","Data":"4fb9ddeebf7fdc48275aa127beae95960332cc7b0a412c68ec9c983dea3dd80f"}
Oct 30 18:01:30.496485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:30.496428    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd" podStartSLOduration=1.4964079 podStartE2EDuration="1.4964079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:30.494841022 +0000 UTC m=+2896.256884836" watchObservedRunningTime="2026-10-30 18:01:30.4964079 +0000 UTC m=+2896.258451715"
Oct 30 18:01:31.467933 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:31.467892    5534 generic.go:356] "Generic (PLEG): container finished" podID="0670fb6b-0681-455e-a46e-b9a18805a1d5" containerID="318ae302e6fef49e500318d69fb324ccc14eeba4e9fa8fa6e97f3515d84b059d" exitCode=0
Oct 30 18:01:31.468444 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:31.468155    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd" event={"ID":"0670fb6b-0681-455e-a46e-b9a18805a1d5","Type":"ContainerDied","Data":"318ae302e6fef49e500318d69fb324ccc14eeba4e9fa8fa6e97f3515d84b059d"}
Oct 30 18:01:31.809048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:31.808947    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:32.658399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.658369    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:32.782803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.782698    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca\") pod \"0670fb6b-0681-455e-a46e-b9a18805a1d5\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") "
Oct 30 18:01:32.782996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.782848    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd\" (UniqueName: \"kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd\") pod \"0670fb6b-0681-455e-a46e-b9a18805a1d5\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") "
Oct 30 18:01:32.782996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.782925    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca\") pod \"0670fb6b-0681-455e-a46e-b9a18805a1d5\" (UID: \"0670fb6b-0681-455e-a46e-b9a18805a1d5\") "
Oct 30 18:01:32.783333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.783301    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca" pod "0670fb6b-0681-455e-a46e-b9a18805a1d5" (UID: "0670fb6b-0681-455e-a46e-b9a18805a1d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:01:32.783935 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.783912    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:32.785016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.784989    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca" pod "0670fb6b-0681-455e-a46e-b9a18805a1d5" (UID: "0670fb6b-0681-455e-a46e-b9a18805a1d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:01:32.785199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.785178    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd" pod "0670fb6b-0681-455e-a46e-b9a18805a1d5" (UID: "0670fb6b-0681-455e-a46e-b9a18805a1d5"). InnerVolumeSpecName "kube-api-access-srjpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:01:32.885719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.885678    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjpd\" (UniqueName: \"kubernetes.io/projected/0670fb6b-0681-455e-a46e-b9a18805a1d5-kube-api-access-srjpd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:32.885719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:32.885711    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0670fb6b-0681-455e-a46e-b9a18805a1d5-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:33.401184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:33.401146    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:33.491444 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:33.491403    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd" event={"ID":"0670fb6b-0681-455e-a46e-b9a18805a1d5","Type":"ContainerDied","Data":"4fb9ddeebf7fdc48275aa127beae95960332cc7b0a412c68ec9c983dea3dd80f"}
Oct 30 18:01:33.491634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:33.491470    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb9ddeebf7fdc48275aa127beae95960332cc7b0a412c68ec9c983dea3dd80f"
Oct 30 18:01:33.491634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:33.491536    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"
Oct 30 18:01:35.070244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.070201    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:35.709730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.709691    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"]
Oct 30 18:01:35.710901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.710879    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0670fb6b-0681-455e-a46e-b9a18805a1d5" containerName="revoke-keys"
Oct 30 18:01:35.710965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.710904    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0670fb6b-0681-455e-a46e-b9a18805a1d5" containerName="revoke-keys"
Oct 30 18:01:35.711118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.711106    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0670fb6b-0681-455e-a46e-b9a18805a1d5" containerName="revoke-keys"
Oct 30 18:01:35.720518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.720478    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.740216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.740185    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"]
Oct 30 18:01:35.746178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.744859    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-90b48a-openshift-default-dockercfg-gwwvs\""
Oct 30 18:01:35.833070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833023    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833088    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833200    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvxn\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833235    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833261    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833279    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833365    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833393    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.833474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.833414    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935324    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935382    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935439    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvxn\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935470    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935501    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935526    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935608    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935662    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.935702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.935693    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.936121 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.936096    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.936582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.936545    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.937081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.937042    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.937275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.937099    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.937411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.937390    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.939056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.939014    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.939154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.939124    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.960272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.960196    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:35.960522 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:35.960272    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvxn\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:36.049195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.049122    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:36.404975 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:01:36.404928    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96da3fe9_965f_46c8_a94a_e9bf4ada3839.slice/crio-2cdeef6bee7d37a40d362f674afd73edbdadf6c97cdede85e2a4373d797dc939 WatchSource:0}: Error finding container 2cdeef6bee7d37a40d362f674afd73edbdadf6c97cdede85e2a4373d797dc939: Status 404 returned error can't find the container with id 2cdeef6bee7d37a40d362f674afd73edbdadf6c97cdede85e2a4373d797dc939
Oct 30 18:01:36.406391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.406362    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"]
Oct 30 18:01:36.409094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.409053    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:36.409210 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.409141    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:36.409210 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.409184    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:36.523534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.522338    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" event={"ID":"96da3fe9-965f-46c8-a94a-e9bf4ada3839","Type":"ContainerStarted","Data":"762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738"}
Oct 30 18:01:36.523534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.522404    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" event={"ID":"96da3fe9-965f-46c8-a94a-e9bf4ada3839","Type":"ContainerStarted","Data":"2cdeef6bee7d37a40d362f674afd73edbdadf6c97cdede85e2a4373d797dc939"}
Oct 30 18:01:36.575693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.575328    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podStartSLOduration=1.575292642 podStartE2EDuration="1.575292642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:36.567811858 +0000 UTC m=+2902.329855674" watchObservedRunningTime="2026-10-30 18:01:36.575292642 +0000 UTC m=+2902.337336456"
Oct 30 18:01:36.896145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:36.896087    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:37.050300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.050265    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:37.051744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.051701    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:37.051893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.051750    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:37.379464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.379432    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"]
Oct 30 18:01:37.384723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.384699    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.401670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.401615    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-738845-openshift-default-dockercfg-5lqwd\""
Oct 30 18:01:37.418374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.418347    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"]
Oct 30 18:01:37.463005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.462976    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463222    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463349    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463439    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463488    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463527    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463603    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463759 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463694    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.463819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.463758    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgbq\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.567880    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgbq\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568009    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568196    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568279    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568350    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568396    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568435    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568498    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.568568    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.569667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.569482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.571084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.571052    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.571726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.571701    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.571974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.571952    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.573925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.572497    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.577694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.576446    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.593667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.578081    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.603677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.603332    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgbq\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.604468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.604420    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:37.701069 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:37.700986    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:38.050257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.050157    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:38.050257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.050218    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:38.093906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.093874    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"]
Oct 30 18:01:38.094551 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:01:38.094522    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d1619a_9925_41f8_a0ec_2e2af4d1bac4.slice/crio-d4b5ad52b71d86c1c0abe38e5ceede48f21e5d5c6596729410c0736b8dcde946 WatchSource:0}: Error finding container d4b5ad52b71d86c1c0abe38e5ceede48f21e5d5c6596729410c0736b8dcde946: Status 404 returned error can't find the container with id d4b5ad52b71d86c1c0abe38e5ceede48f21e5d5c6596729410c0736b8dcde946
Oct 30 18:01:38.098106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.098064    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:38.098210 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.098157    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:38.098210 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.098201    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:01:38.547055 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.547016    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" event={"ID":"64d1619a-9925-41f8-a0ec-2e2af4d1bac4","Type":"ContainerStarted","Data":"48bb2676d5a0c63f714434762114b5958b79db1511400eae3db7245d699bd3d3"}
Oct 30 18:01:38.547589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.547078    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" event={"ID":"64d1619a-9925-41f8-a0ec-2e2af4d1bac4","Type":"ContainerStarted","Data":"d4b5ad52b71d86c1c0abe38e5ceede48f21e5d5c6596729410c0736b8dcde946"}
Oct 30 18:01:38.596329 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.595828    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podStartSLOduration=1.5958079170000001 podStartE2EDuration="1.595807917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:38.595064207 +0000 UTC m=+2904.357108020" watchObservedRunningTime="2026-10-30 18:01:38.595807917 +0000 UTC m=+2904.357851731"
Oct 30 18:01:38.636096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.636066    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:38.705666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:38.704724    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:39.049457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.049426    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:39.049653 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.049478    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:39.377669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.370468    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"]
Oct 30 18:01:39.394687 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.394067    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.394687 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.394351    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"]
Oct 30 18:01:39.414490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.414465    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.414828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.414802    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.415120 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.415092    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwvm\" (UniqueName: \"kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.421125 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.421103    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:01:39.421220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.421112    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:01:39.516254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.516208    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwvm\" (UniqueName: \"kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.516462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.516291    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.516462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.516334    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.517180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.517160    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.519019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.519000    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.539694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.539671    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwvm\" (UniqueName: \"kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm\") pod \"maas-api-revoke-sub-38255b35-cmj8v\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:39.708720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.708110    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:39.708720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.708194    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:39.721504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:39.721464    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:40.050697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.050576    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:40.050697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.050661    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:40.131670 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:01:40.127402    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebb47eb_d9bb_4147_9b45_3e24bf5058dd.slice/crio-0f4690987c99169b756985423fca6a91a2b7c00a5e3403fab15072dc6d8b1ed9 WatchSource:0}: Error finding container 0f4690987c99169b756985423fca6a91a2b7c00a5e3403fab15072dc6d8b1ed9: Status 404 returned error can't find the container with id 0f4690987c99169b756985423fca6a91a2b7c00a5e3403fab15072dc6d8b1ed9
Oct 30 18:01:40.133290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.133254    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"]
Oct 30 18:01:40.619666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.604152    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:40.619666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.613394    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v" event={"ID":"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd","Type":"ContainerStarted","Data":"0f4690987c99169b756985423fca6a91a2b7c00a5e3403fab15072dc6d8b1ed9"}
Oct 30 18:01:40.714741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.710764    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:40.714741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:40.710827    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:41.058933 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:41.053627    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:41.058933 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:41.053711    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:41.636672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:41.636231    5534 generic.go:356] "Generic (PLEG): container finished" podID="6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" containerID="df0651d029d0bf2c7343cbc67512894ffce64cd017917508850c175e75346fc7" exitCode=0
Oct 30 18:01:41.636672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:41.636387    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v" event={"ID":"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd","Type":"ContainerDied","Data":"df0651d029d0bf2c7343cbc67512894ffce64cd017917508850c175e75346fc7"}
Oct 30 18:01:41.707731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:41.707543    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:41.707731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:41.707602    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:42.050030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:42.049940    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:42.050030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:42.050009    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:42.703932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:42.703898    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:42.704631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:42.704602    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:42.735469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:42.735441    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:43.050185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.050087    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 18:01:43.050793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.050761    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 18:01:43.190716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.187881    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:43.338685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.338128    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm\" (UniqueName: \"kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm\") pod \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") "
Oct 30 18:01:43.338685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.338550    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca\") pod \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") "
Oct 30 18:01:43.338685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.338600    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca\") pod \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\" (UID: \"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd\") "
Oct 30 18:01:43.339333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.339309    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca" pod "6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" (UID: "6ebb47eb-d9bb-4147-9b45-3e24bf5058dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:01:43.339453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.339434    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:43.341008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.340977    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm" pod "6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" (UID: "6ebb47eb-d9bb-4147-9b45-3e24bf5058dd"). InnerVolumeSpecName "kube-api-access-prwvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:01:43.341593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.341562    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca" pod "6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" (UID: "6ebb47eb-d9bb-4147-9b45-3e24bf5058dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:01:43.441187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.441151    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:43.441187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.441184    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prwvm\" (UniqueName: \"kubernetes.io/projected/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd-kube-api-access-prwvm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:01:43.670701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.666503    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v" event={"ID":"6ebb47eb-d9bb-4147-9b45-3e24bf5058dd","Type":"ContainerDied","Data":"0f4690987c99169b756985423fca6a91a2b7c00a5e3403fab15072dc6d8b1ed9"}
Oct 30 18:01:43.670701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.666566    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4690987c99169b756985423fca6a91a2b7c00a5e3403fab15072dc6d8b1ed9"
Oct 30 18:01:43.670701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.666765    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"
Oct 30 18:01:43.701576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.701543    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:43.701906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:43.701875    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:44.066727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:44.066451    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:44.070666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:44.068145    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:44.074798 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:44.071878    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:01:44.693595 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:44.693557    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:44.721400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:44.721367    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:44.721739 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:44.721706    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:45.250187 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:01:45.250141    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:01:45.702364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:45.702329    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 18:01:45.702733 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:45.702391    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 18:01:46.708216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:46.708185    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:46.708827 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:46.708565    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:46.709896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:46.709874    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:01:47.011771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:47.011683    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:48.667149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:48.667112    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:48.781233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:48.781204    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:50.388600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:50.388565    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:52.080201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:52.080162    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:53.430568 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:53.430534    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:53.800036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:53.799955    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:55.675295 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:55.675265    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:56.961147 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:56.961117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:57.193936 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:57.193902    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:58.075045 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:58.075001    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:01:59.912341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:01:59.912310    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:01.543557 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:01.543524    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:02.291960 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:02.291923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:03.452502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:03.452469    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"]
Oct 30 18:02:03.503078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:03.503034    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:03.626691 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:03.626605    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:05.215593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:05.215511    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:07.036839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:07.036802    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:07.149152 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:07.149117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:07.281690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:07.277673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:08.452903 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.452852    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944" podUID="9c09984f-edfd-4b00-b553-b328608173d2" containerName="payload-pre-processing" containerID="cri-o://6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a" gracePeriod=25
Oct 30 18:02:08.775589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.775547    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:02:08.809922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.809891    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:08.813243 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.813216    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn\") pod \"9c09984f-edfd-4b00-b553-b328608173d2\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") "
Oct 30 18:02:08.813357 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.813309    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume\") pod \"9c09984f-edfd-4b00-b553-b328608173d2\" (UID: \"9c09984f-edfd-4b00-b553-b328608173d2\") "
Oct 30 18:02:08.813730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.813704    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume" pod "9c09984f-edfd-4b00-b553-b328608173d2" (UID: "9c09984f-edfd-4b00-b553-b328608173d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:02:08.814128 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.814107    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c09984f-edfd-4b00-b553-b328608173d2-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:08.815908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.815848    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn" pod "9c09984f-edfd-4b00-b553-b328608173d2" (UID: "9c09984f-edfd-4b00-b553-b328608173d2"). InnerVolumeSpecName "kube-api-access-jt5jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:08.916434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:08.916401    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/9c09984f-edfd-4b00-b553-b328608173d2-kube-api-access-jt5jn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:09.068160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.068010    5534 generic.go:356] "Generic (PLEG): container finished" podID="9c09984f-edfd-4b00-b553-b328608173d2" containerID="6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a" exitCode=0
Oct 30 18:02:09.068160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.068053    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"
Oct 30 18:02:09.068160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.068098    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944" event={"ID":"9c09984f-edfd-4b00-b553-b328608173d2","Type":"ContainerDied","Data":"6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a"}
Oct 30 18:02:09.068468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.068175    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944" event={"ID":"9c09984f-edfd-4b00-b553-b328608173d2","Type":"ContainerDied","Data":"7e5d9ce165ae9f9a47c662dab6047f368c6333b976e1d5a8a9ce7bb0d6d6104c"}
Oct 30 18:02:09.068468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.068210    5534 scope.go:122] "RemoveContainer" containerID="6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a"
Oct 30 18:02:09.081568 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.081506    5534 scope.go:122] "RemoveContainer" containerID="6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a"
Oct 30 18:02:09.081850 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:02:09.081828    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a\": container with ID starting with 6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a not found: ID does not exist" containerID="6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a"
Oct 30 18:02:09.081920 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.081864    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a"} err="failed to get container status \"6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a\": rpc error: code = NotFound desc = could not find container \"6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a\": container with ID starting with 6c1d117acefb27cde06fd6b3b5259d081024ec28b919718b1da47d834580b03a not found: ID does not exist"
Oct 30 18:02:09.124048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.124019    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"]
Oct 30 18:02:09.134843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:09.134821    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-7cc7849f9f-kp944"]
Oct 30 18:02:10.372561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:10.372523    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c09984f-edfd-4b00-b553-b328608173d2" path="/var/lib/kubelet/pods/9c09984f-edfd-4b00-b553-b328608173d2/volumes"
Oct 30 18:02:10.611662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:10.611615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:12.333317 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:12.333282    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:14.635149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.635119    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"]
Oct 30 18:02:14.636005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.635988    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" containerName="revoke-keys"
Oct 30 18:02:14.636069 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.636007    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" containerName="revoke-keys"
Oct 30 18:02:14.636069 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.636040    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9c09984f-edfd-4b00-b553-b328608173d2" containerName="payload-pre-processing"
Oct 30 18:02:14.636069 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.636045    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c09984f-edfd-4b00-b553-b328608173d2" containerName="payload-pre-processing"
Oct 30 18:02:14.636168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.636148    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c09984f-edfd-4b00-b553-b328608173d2" containerName="payload-pre-processing"
Oct 30 18:02:14.636168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.636164    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" containerName="revoke-keys"
Oct 30 18:02:14.640553 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.640530    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.665398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.665366    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"]
Oct 30 18:02:14.675874 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.675842    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:02:14.676040 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.675995    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:02:14.796079 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.796044    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:14.811909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.811872    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68n8\" (UniqueName: \"kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.812116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.812062    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.812174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.812151    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.914092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.913978    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.914092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.914054    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.914325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.914302    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68n8\" (UniqueName: \"kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.914959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.914932    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.916610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.916589    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.950840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.938308    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68n8\" (UniqueName: \"kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8\") pod \"maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:14.967513 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:14.967478    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:15.247503 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:02:15.247422    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:02:15.290665 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:02:15.290602    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14ace23_27ce_4616_978e_73c6f214aa97.slice/crio-4bd1624651d5cd06c81b4d249ca2987902b151420b087d127cdce01eba7fc355 WatchSource:0}: Error finding container 4bd1624651d5cd06c81b4d249ca2987902b151420b087d127cdce01eba7fc355: Status 404 returned error can't find the container with id 4bd1624651d5cd06c81b4d249ca2987902b151420b087d127cdce01eba7fc355
Oct 30 18:02:15.292600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:15.292576    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"]
Oct 30 18:02:16.135167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:16.135129    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c" event={"ID":"d14ace23-27ce-4616-978e-73c6f214aa97","Type":"ContainerStarted","Data":"235913b806e3dba04cc1164b22e2f866da6e2b99f5b9939a2e71e4472d13e85e"}
Oct 30 18:02:16.135564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:16.135197    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c" event={"ID":"d14ace23-27ce-4616-978e-73c6f214aa97","Type":"ContainerStarted","Data":"4bd1624651d5cd06c81b4d249ca2987902b151420b087d127cdce01eba7fc355"}
Oct 30 18:02:16.177858 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:16.177799    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c" podStartSLOduration=2.17777985 podStartE2EDuration="2.17777985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:16.175549957 +0000 UTC m=+2941.937593773" watchObservedRunningTime="2026-10-30 18:02:16.17777985 +0000 UTC m=+2941.939823664"
Oct 30 18:02:16.720018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:16.719985    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:17.146503 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:17.146460    5534 generic.go:356] "Generic (PLEG): container finished" podID="d14ace23-27ce-4616-978e-73c6f214aa97" containerID="235913b806e3dba04cc1164b22e2f866da6e2b99f5b9939a2e71e4472d13e85e" exitCode=0
Oct 30 18:02:17.147070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:17.146815    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c" event={"ID":"d14ace23-27ce-4616-978e-73c6f214aa97","Type":"ContainerDied","Data":"235913b806e3dba04cc1164b22e2f866da6e2b99f5b9939a2e71e4472d13e85e"}
Oct 30 18:02:18.035496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.035463    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"]
Oct 30 18:02:18.044622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.044594    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb8f8e3-8zzr6"]
Oct 30 18:02:18.335859 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.335834    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:18.372712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.372682    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8feb148-6af9-4a31-a4e2-d90d1b2c0592" path="/var/lib/kubelet/pods/f8feb148-6af9-4a31-a4e2-d90d1b2c0592/volumes"
Oct 30 18:02:18.480447 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.480420    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca\") pod \"d14ace23-27ce-4616-978e-73c6f214aa97\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") "
Oct 30 18:02:18.480664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.480469    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8\" (UniqueName: \"kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8\") pod \"d14ace23-27ce-4616-978e-73c6f214aa97\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") "
Oct 30 18:02:18.480664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.480568    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca\") pod \"d14ace23-27ce-4616-978e-73c6f214aa97\" (UID: \"d14ace23-27ce-4616-978e-73c6f214aa97\") "
Oct 30 18:02:18.480987 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.480962    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca" pod "d14ace23-27ce-4616-978e-73c6f214aa97" (UID: "d14ace23-27ce-4616-978e-73c6f214aa97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:02:18.481396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.481367    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:18.482821 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.482797    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca" pod "d14ace23-27ce-4616-978e-73c6f214aa97" (UID: "d14ace23-27ce-4616-978e-73c6f214aa97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:02:18.482942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.482855    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8" pod "d14ace23-27ce-4616-978e-73c6f214aa97" (UID: "d14ace23-27ce-4616-978e-73c6f214aa97"). InnerVolumeSpecName "kube-api-access-r68n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:18.583062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.582985    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14ace23-27ce-4616-978e-73c6f214aa97-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:18.583062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.583011    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68n8\" (UniqueName: \"kubernetes.io/projected/d14ace23-27ce-4616-978e-73c6f214aa97-kube-api-access-r68n8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:18.606405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:18.606371    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:19.165021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:19.164994    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"
Oct 30 18:02:19.165203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:19.165023    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c" event={"ID":"d14ace23-27ce-4616-978e-73c6f214aa97","Type":"ContainerDied","Data":"4bd1624651d5cd06c81b4d249ca2987902b151420b087d127cdce01eba7fc355"}
Oct 30 18:02:19.165203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:19.165096    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd1624651d5cd06c81b4d249ca2987902b151420b087d127cdce01eba7fc355"
Oct 30 18:02:19.966057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:19.966020    5534 scope.go:122] "RemoveContainer" containerID="b40af43b22c7735f8914b8438d8053669db8b56ee1811cf57e92a09fb566df1c"
Oct 30 18:02:19.981867 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:19.980816    5534 scope.go:122] "RemoveContainer" containerID="cd7d432784c41c691349ac3c5f5caebdc52f2d92f58761d8049ec2d7e5490049"
Oct 30 18:02:19.994784 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:19.994764    5534 scope.go:122] "RemoveContainer" containerID="bb6db243a709860c805fd054b9fbc525005dd9826a0d70d46f8079301fa05208"
Oct 30 18:02:20.450657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:20.450598    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:21.434623 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:21.434593    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"]
Oct 30 18:02:21.573052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:21.572971    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:21.693756 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:21.693615    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:21.824331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:21.824296    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:22.685566 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:22.685527    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:24.328322 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:24.328279    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:26.105485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.105448    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:26.435499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.435412    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v" podUID="c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" containerName="payload-processing" containerID="cri-o://54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c" gracePeriod=25
Oct 30 18:02:26.744608 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.744573    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:02:26.830296 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.830265    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch\" (UniqueName: \"kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch\") pod \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") "
Oct 30 18:02:26.830495 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.830312    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume\") pod \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\" (UID: \"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c\") "
Oct 30 18:02:26.830788 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.830761    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume" pod "c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" (UID: "c7db7158-d21c-4e36-af1d-27aa6b4dfe8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:02:26.831028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.831011    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:26.832714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.832678    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch" pod "c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" (UID: "c7db7158-d21c-4e36-af1d-27aa6b4dfe8c"). InnerVolumeSpecName "kube-api-access-mntch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:26.932257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:26.932224    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntch\" (UniqueName: \"kubernetes.io/projected/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c-kube-api-access-mntch\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:27.261208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.261171    5534 generic.go:356] "Generic (PLEG): container finished" podID="c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" containerID="54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c" exitCode=0
Oct 30 18:02:27.261675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.261227    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"
Oct 30 18:02:27.261675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.261255    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v" event={"ID":"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c","Type":"ContainerDied","Data":"54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c"}
Oct 30 18:02:27.261675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.261317    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v" event={"ID":"c7db7158-d21c-4e36-af1d-27aa6b4dfe8c","Type":"ContainerDied","Data":"e5a872841efeb72753d2ebadc91ae2f9dbb3dfe1513d12a5f3c1fc25b1d9cb01"}
Oct 30 18:02:27.261675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.261348    5534 scope.go:122] "RemoveContainer" containerID="54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c"
Oct 30 18:02:27.274839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.274812    5534 scope.go:122] "RemoveContainer" containerID="54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c"
Oct 30 18:02:27.275177 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:02:27.275150    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c\": container with ID starting with 54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c not found: ID does not exist" containerID="54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c"
Oct 30 18:02:27.275256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.275188    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c"} err="failed to get container status \"54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c\": rpc error: code = NotFound desc = could not find container \"54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c\": container with ID starting with 54afac6ab952a1f7ffac6d0e44f54b52abbf5a8adc2816ce02818031509db87c not found: ID does not exist"
Oct 30 18:02:27.318856 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.318829    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"]
Oct 30 18:02:27.325081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.325043    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-7665669778-7ql2v"]
Oct 30 18:02:27.374311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.374280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:27.502836 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.502785    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:27.833931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:27.833896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:28.034858 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:28.034819    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:28.375537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:28.375496    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" path="/var/lib/kubelet/pods/c7db7158-d21c-4e36-af1d-27aa6b4dfe8c/volumes"
Oct 30 18:02:28.526365 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:28.526334    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:28.871824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:28.871776    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:29.041162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:29.041128    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"]
Oct 30 18:02:29.045123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:29.045100    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-055514d9-7b2j4"]
Oct 30 18:02:29.592780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:29.592741    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:30.375470 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:30.375429    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b6c791-b042-4d23-8864-3f3637848091" path="/var/lib/kubelet/pods/22b6c791-b042-4d23-8864-3f3637848091/volumes"
Oct 30 18:02:31.377969 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:31.377935    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:32.676062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:32.676028    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:32.820200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:32.820166    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:33.301014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:33.300975    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:34.347530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.347492    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:34.433690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.433660    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"]
Oct 30 18:02:34.434612 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.434591    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" containerName="payload-processing"
Oct 30 18:02:34.434612 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.434613    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" containerName="payload-processing"
Oct 30 18:02:34.434780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.434667    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d14ace23-27ce-4616-978e-73c6f214aa97" containerName="revoke-keys"
Oct 30 18:02:34.434780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.434675    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14ace23-27ce-4616-978e-73c6f214aa97" containerName="revoke-keys"
Oct 30 18:02:34.434880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.434864    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7db7158-d21c-4e36-af1d-27aa6b4dfe8c" containerName="payload-processing"
Oct 30 18:02:34.434937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.434894    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14ace23-27ce-4616-978e-73c6f214aa97" containerName="revoke-keys"
Oct 30 18:02:34.440238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.440209    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.462770 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.462741    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"]
Oct 30 18:02:34.468991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.468967    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95f55d\""
Oct 30 18:02:34.477728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.477700    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:34.478883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.478856    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfb8w\" (UniqueName: \"kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w\") pod \"payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.479718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.479693    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.512978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.512948    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"]
Oct 30 18:02:34.520303 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.520282    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:34.534089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.534066    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"]
Oct 30 18:02:34.535023 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.535004    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95f55d-dockercfg-nfxlk\""
Oct 30 18:02:34.588665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.586462    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.588665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.586726    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:34.588665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.586810    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfb8w\" (UniqueName: \"kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w\") pod \"payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.588665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.586934    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dprq\" (UniqueName: \"kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:34.634717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.634143    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.688219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.688176    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:34.771533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.768018    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:34.772483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.771974    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dprq\" (UniqueName: \"kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:34.846662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.846601    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfb8w\" (UniqueName: \"kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w\") pod \"payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:34.864939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:34.864893    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dprq\" (UniqueName: \"kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:35.059289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:35.059203    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:35.139264 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:35.139057    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:35.149812 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:35.149467    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:35.399118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:35.399082    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"]
Oct 30 18:02:35.534341 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:02:35.534309    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba9a1be_9936_47a6_9479_00e5aa827fc8.slice/crio-323c41f04bbf14ff29e5778356e529d04f9a52a2e047d20da1c19bf33cb728da WatchSource:0}: Error finding container 323c41f04bbf14ff29e5778356e529d04f9a52a2e047d20da1c19bf33cb728da: Status 404 returned error can't find the container with id 323c41f04bbf14ff29e5778356e529d04f9a52a2e047d20da1c19bf33cb728da
Oct 30 18:02:35.534924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:35.534877    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"]
Oct 30 18:02:36.349266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.349228    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz" event={"ID":"deb7cb4b-06b9-4d0a-9b44-be8addc82864","Type":"ContainerStarted","Data":"47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12"}
Oct 30 18:02:36.349459 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.349321    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz" event={"ID":"deb7cb4b-06b9-4d0a-9b44-be8addc82864","Type":"ContainerStarted","Data":"e3ccf606f912e90a3b96b2a718ad7ffba904a16580ef89fafa4d086cf99d2d7d"}
Oct 30 18:02:36.349537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.349519    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:36.350837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.350810    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659" event={"ID":"cba9a1be-9936-47a6-9479-00e5aa827fc8","Type":"ContainerStarted","Data":"db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52"}
Oct 30 18:02:36.350945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.350856    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659" event={"ID":"cba9a1be-9936-47a6-9479-00e5aa827fc8","Type":"ContainerStarted","Data":"323c41f04bbf14ff29e5778356e529d04f9a52a2e047d20da1c19bf33cb728da"}
Oct 30 18:02:36.350945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.350881    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:36.391166 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.391112    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz" podStartSLOduration=2.3910993019999998 podStartE2EDuration="2.391099302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:36.388732009 +0000 UTC m=+2962.150775822" watchObservedRunningTime="2026-10-30 18:02:36.391099302 +0000 UTC m=+2962.153143116"
Oct 30 18:02:36.431803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.431758    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659" podStartSLOduration=2.431745537 podStartE2EDuration="2.431745537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:36.427920126 +0000 UTC m=+2962.189963939" watchObservedRunningTime="2026-10-30 18:02:36.431745537 +0000 UTC m=+2962.193789349"
Oct 30 18:02:36.909212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:36.909175    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:38.534938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:38.534884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:39.605062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:39.605025    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"]
Oct 30 18:02:39.605689 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:39.605272    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy" containerID="cri-o://5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335" gracePeriod=30
Oct 30 18:02:40.848234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:40.848156    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:42.822872 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:42.822836    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:44.127521 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.127488    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"]
Oct 30 18:02:44.132133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.132114    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.167254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.167228    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:02:44.167254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.167240    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:02:44.186036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.186011    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"]
Oct 30 18:02:44.283461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.283427    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.283612 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.283595    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.283726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.283712    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgzs\" (UniqueName: \"kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.385577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.385499    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.385771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.385589    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgzs\" (UniqueName: \"kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.385837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.385779    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.386382 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.386359    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.388089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.388067    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.409250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.409221    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgzs\" (UniqueName: \"kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs\") pod \"maas-api-revoke-sub-781bc6ac-jw886\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.448526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.448501    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:44.619264 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.619221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:44.774431 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:02:44.774397    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c03e3fa_0223_4985_8d0e_898e13445b72.slice/crio-a2ce405a63e9e11b7c855a1a71ff298890286e4533cf60550b52c212f197a04a WatchSource:0}: Error finding container a2ce405a63e9e11b7c855a1a71ff298890286e4533cf60550b52c212f197a04a: Status 404 returned error can't find the container with id a2ce405a63e9e11b7c855a1a71ff298890286e4533cf60550b52c212f197a04a
Oct 30 18:02:44.775416 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.775350    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"]
Oct 30 18:02:44.983177 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:44.983151    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 18:02:45.096388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096348    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.096573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096467    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.096573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096516    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.096728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096581    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.096728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096673    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.096728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096707    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.097118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096839    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:02:45.097118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096923    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:02:45.097318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.097180    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.097318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.096964    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:02:45.097318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.097240    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.097318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.097290    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy\") pod \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\" (UID: \"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a\") "
Oct 30 18:02:45.097536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.097458    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:02:45.097536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.097013    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:02:45.098254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.098226    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.098388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.098258    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.098388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.098276    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.098388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.098291    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.098388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.098306    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.099201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.099169    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "kube-api-access-zjfnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:45.099364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.099345    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:02:45.099926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.099908    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:45.100385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.100362    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy" pod "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" (UID: "d98ef006-a19a-49c4-b9a3-6f9a327d1b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:02:45.200069 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.200039    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfnd\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-kube-api-access-zjfnd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.200419 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.200073    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.200419 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.200088    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.200419 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.200103    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:45.245372 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:02:45.245292    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:02:45.446024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.445987    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886" event={"ID":"8c03e3fa-0223-4985-8d0e-898e13445b72","Type":"ContainerStarted","Data":"e48af2a68706c29a87aa8b008003a66ad773cf208a938d4808fafa16ecac3060"}
Oct 30 18:02:45.446176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.446053    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886" event={"ID":"8c03e3fa-0223-4985-8d0e-898e13445b72","Type":"ContainerStarted","Data":"a2ce405a63e9e11b7c855a1a71ff298890286e4533cf60550b52c212f197a04a"}
Oct 30 18:02:45.447460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.447433    5534 generic.go:356] "Generic (PLEG): container finished" podID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerID="5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335" exitCode=0
Oct 30 18:02:45.447607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.447470    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"
Oct 30 18:02:45.447607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.447510    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" event={"ID":"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a","Type":"ContainerDied","Data":"5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335"}
Oct 30 18:02:45.447607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.447568    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9" event={"ID":"d98ef006-a19a-49c4-b9a3-6f9a327d1b1a","Type":"ContainerDied","Data":"e2ed22870d495baa8db5b45fbe95a136bce381d5e1aca8e913b869aaf63fe3e7"}
Oct 30 18:02:45.447607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.447596    5534 scope.go:122] "RemoveContainer" containerID="5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335"
Oct 30 18:02:45.461212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.460992    5534 scope.go:122] "RemoveContainer" containerID="5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335"
Oct 30 18:02:45.461331 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:02:45.461310    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335\": container with ID starting with 5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335 not found: ID does not exist" containerID="5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335"
Oct 30 18:02:45.461391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.461338    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335"} err="failed to get container status \"5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335\": rpc error: code = NotFound desc = could not find container \"5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335\": container with ID starting with 5c29d32eb07974f3d73c188226ff20a0ae7327176ec851afcf62ae8a04e25335 not found: ID does not exist"
Oct 30 18:02:45.493295 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.493253    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886" podStartSLOduration=1.493241268 podStartE2EDuration="1.493241268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:45.489186392 +0000 UTC m=+2971.251230208" watchObservedRunningTime="2026-10-30 18:02:45.493241268 +0000 UTC m=+2971.255285102"
Oct 30 18:02:45.530943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.530870    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"]
Oct 30 18:02:45.539865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:45.539842    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-d8pb9"]
Oct 30 18:02:46.037385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.037353    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"]
Oct 30 18:02:46.044185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.044151    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d526a864-gvvpd"]
Oct 30 18:02:46.375404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.375369    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b046dd1-e9c4-4362-8daa-818997c428b8" path="/var/lib/kubelet/pods/4b046dd1-e9c4-4362-8daa-818997c428b8/volumes"
Oct 30 18:02:46.376102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.376084    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" path="/var/lib/kubelet/pods/d98ef006-a19a-49c4-b9a3-6f9a327d1b1a/volumes"
Oct 30 18:02:46.406666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.406438    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:46.470688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.470565    5534 generic.go:356] "Generic (PLEG): container finished" podID="8c03e3fa-0223-4985-8d0e-898e13445b72" containerID="e48af2a68706c29a87aa8b008003a66ad773cf208a938d4808fafa16ecac3060" exitCode=0
Oct 30 18:02:46.470913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:46.470802    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886" event={"ID":"8c03e3fa-0223-4985-8d0e-898e13445b72","Type":"ContainerDied","Data":"e48af2a68706c29a87aa8b008003a66ad773cf208a938d4808fafa16ecac3060"}
Oct 30 18:02:47.362235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.362203    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:02:47.362474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.362439    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:02:47.686430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.686404    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:47.741940 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.741902    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs\" (UniqueName: \"kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs\") pod \"8c03e3fa-0223-4985-8d0e-898e13445b72\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") "
Oct 30 18:02:47.742238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.742215    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca\") pod \"8c03e3fa-0223-4985-8d0e-898e13445b72\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") "
Oct 30 18:02:47.742410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.742386    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca\") pod \"8c03e3fa-0223-4985-8d0e-898e13445b72\" (UID: \"8c03e3fa-0223-4985-8d0e-898e13445b72\") "
Oct 30 18:02:47.742718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.742689    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca" pod "8c03e3fa-0223-4985-8d0e-898e13445b72" (UID: "8c03e3fa-0223-4985-8d0e-898e13445b72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:02:47.743326 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.743301    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:47.744754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.744726    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs" pod "8c03e3fa-0223-4985-8d0e-898e13445b72" (UID: "8c03e3fa-0223-4985-8d0e-898e13445b72"). InnerVolumeSpecName "kube-api-access-8jgzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:47.749570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.745452    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca" pod "8c03e3fa-0223-4985-8d0e-898e13445b72" (UID: "8c03e3fa-0223-4985-8d0e-898e13445b72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:02:47.844520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.844488    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c03e3fa-0223-4985-8d0e-898e13445b72-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:47.844520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:47.844515    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgzs\" (UniqueName: \"kubernetes.io/projected/8c03e3fa-0223-4985-8d0e-898e13445b72-kube-api-access-8jgzs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:48.305015 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:48.304985    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:48.493375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:48.493343    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"
Oct 30 18:02:48.493585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:48.493375    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886" event={"ID":"8c03e3fa-0223-4985-8d0e-898e13445b72","Type":"ContainerDied","Data":"a2ce405a63e9e11b7c855a1a71ff298890286e4533cf60550b52c212f197a04a"}
Oct 30 18:02:48.493585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:48.493428    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2ce405a63e9e11b7c855a1a71ff298890286e4533cf60550b52c212f197a04a"
Oct 30 18:02:50.071463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:50.071422    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:52.467866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:52.467829    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:53.560338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:53.560303    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:54.173300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.173258    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:54.443497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.443374    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"]
Oct 30 18:02:54.444315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.444295    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy"
Oct 30 18:02:54.444315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.444315    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy"
Oct 30 18:02:54.444507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.444377    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8c03e3fa-0223-4985-8d0e-898e13445b72" containerName="revoke-keys"
Oct 30 18:02:54.444507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.444384    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c03e3fa-0223-4985-8d0e-898e13445b72" containerName="revoke-keys"
Oct 30 18:02:54.444610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.444526    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c03e3fa-0223-4985-8d0e-898e13445b72" containerName="revoke-keys"
Oct 30 18:02:54.444610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.444544    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98ef006-a19a-49c4-b9a3-6f9a327d1b1a" containerName="istio-proxy"
Oct 30 18:02:54.448721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.448702    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.484695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.476105    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:02:54.484695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.476839    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:02:54.484695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.483205    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"]
Oct 30 18:02:54.561407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.561363    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb6k2\" (UniqueName: \"kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.561954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.561418    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.561954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.561539    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.663795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.663763    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6k2\" (UniqueName: \"kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.663795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.663799    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.664083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.663857    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.664656 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.664626    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.667085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.667062    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.687161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.687141    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb6k2\" (UniqueName: \"kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2\") pod \"maas-api-revoke-sub-cfafb6be-mpz5x\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:54.701440 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.701354    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:54.770812 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:54.770771    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:55.086633 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:02:55.086592    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b68ca58_f4f0_4cc5_9d26_9cff202ef0ec.slice/crio-425b821091a046dff80e6f4b705feaf4eaa0b0f983f0f2bf7d1ccdecead649fc WatchSource:0}: Error finding container 425b821091a046dff80e6f4b705feaf4eaa0b0f983f0f2bf7d1ccdecead649fc: Status 404 returned error can't find the container with id 425b821091a046dff80e6f4b705feaf4eaa0b0f983f0f2bf7d1ccdecead649fc
Oct 30 18:02:55.088123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.088095    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"]
Oct 30 18:02:55.400128 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.400098    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:55.607762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.607667    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:55.644663 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.642790    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x" event={"ID":"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec","Type":"ContainerStarted","Data":"2765bac4f7f80fc4ca18836fe653ef95ddc0e22ac051790cfd0df9d4366cd0ff"}
Oct 30 18:02:55.644663 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.642854    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x" event={"ID":"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec","Type":"ContainerStarted","Data":"425b821091a046dff80e6f4b705feaf4eaa0b0f983f0f2bf7d1ccdecead649fc"}
Oct 30 18:02:55.686849 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.686753    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x" podStartSLOduration=1.686738847 podStartE2EDuration="1.686738847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:55.685167264 +0000 UTC m=+2981.447211107" watchObservedRunningTime="2026-10-30 18:02:55.686738847 +0000 UTC m=+2981.448782661"
Oct 30 18:02:55.735178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.735142    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:55.871669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.867750    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"]
Oct 30 18:02:55.984731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.911341    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"]
Oct 30 18:02:55.984731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.911461    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:55.984731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.968147    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-584a12\""
Oct 30 18:02:55.984731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.973065    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-584a12-dockercfg-2txsp\""
Oct 30 18:02:55.984731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.980901    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"]
Oct 30 18:02:55.993042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.987186    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:55.997502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.997062    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:55.997502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:55.997138    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfddl\" (UniqueName: \"kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl\") pod \"payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:56.022877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.022135    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"]
Oct 30 18:02:56.048744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.048532    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"]
Oct 30 18:02:56.052564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.052529    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fc7e49-wzgsw"]
Oct 30 18:02:56.099144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.099107    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:56.099318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.099167    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfddl\" (UniqueName: \"kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl\") pod \"payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:56.099318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.099194    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbbm\" (UniqueName: \"kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm\") pod \"payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.099459 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.099331    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.100062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.100036    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:56.124115 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.124084    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfddl\" (UniqueName: \"kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl\") pod \"payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:56.201444 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.201410    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbbm\" (UniqueName: \"kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm\") pod \"payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.201592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.201575    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.202513 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.202485    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.224589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.224563    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbbm\" (UniqueName: \"kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm\") pod \"payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.319393 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.319319    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:56.322187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.322164    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:56.377662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.377458    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8f063e-7038-4035-b14e-e2519fa2ec43" path="/var/lib/kubelet/pods/8b8f063e-7038-4035-b14e-e2519fa2ec43/volumes"
Oct 30 18:02:56.548380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.548345    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:56.654970 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.654936    5534 generic.go:356] "Generic (PLEG): container finished" podID="8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" containerID="2765bac4f7f80fc4ca18836fe653ef95ddc0e22ac051790cfd0df9d4366cd0ff" exitCode=0
Oct 30 18:02:56.655469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.654994    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x" event={"ID":"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec","Type":"ContainerDied","Data":"2765bac4f7f80fc4ca18836fe653ef95ddc0e22ac051790cfd0df9d4366cd0ff"}
Oct 30 18:02:56.660078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.660051    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"]
Oct 30 18:02:56.660487 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:02:56.660457    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25abd8f_0f00_4e2e_a31d_9964689d83c2.slice/crio-4a757b2755566a364d0df87314944377f70d749c868029dd40e623c45e9f2792 WatchSource:0}: Error finding container 4a757b2755566a364d0df87314944377f70d749c868029dd40e623c45e9f2792: Status 404 returned error can't find the container with id 4a757b2755566a364d0df87314944377f70d749c868029dd40e623c45e9f2792
Oct 30 18:02:56.697371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:56.697337    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"]
Oct 30 18:02:56.699974 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:02:56.699945    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8c21f3_9028_4e39_a7e3_f4de43083a67.slice/crio-ae31bcff760d7e2ebf3f87d8fb2839626c14988db3048412f7243065e18b5026 WatchSource:0}: Error finding container ae31bcff760d7e2ebf3f87d8fb2839626c14988db3048412f7243065e18b5026: Status 404 returned error can't find the container with id ae31bcff760d7e2ebf3f87d8fb2839626c14988db3048412f7243065e18b5026
Oct 30 18:02:57.667969 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.667914    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p" event={"ID":"ce8c21f3-9028-4e39-a7e3-f4de43083a67","Type":"ContainerStarted","Data":"a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab"}
Oct 30 18:02:57.668514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.668016    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p" event={"ID":"ce8c21f3-9028-4e39-a7e3-f4de43083a67","Type":"ContainerStarted","Data":"ae31bcff760d7e2ebf3f87d8fb2839626c14988db3048412f7243065e18b5026"}
Oct 30 18:02:57.668514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.668049    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:02:57.670209 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.670173    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr" event={"ID":"b25abd8f-0f00-4e2e-a31d-9964689d83c2","Type":"ContainerStarted","Data":"ed26351f30c16fdfc7c295f6b4a62e4ad253904197fe8f7bd178dbd1aedcb9d0"}
Oct 30 18:02:57.670364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.670258    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr" event={"ID":"b25abd8f-0f00-4e2e-a31d-9964689d83c2","Type":"ContainerStarted","Data":"4a757b2755566a364d0df87314944377f70d749c868029dd40e623c45e9f2792"}
Oct 30 18:02:57.670364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.670320    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:02:57.716308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.716238    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p" podStartSLOduration=2.716218569 podStartE2EDuration="2.716218569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:57.710915594 +0000 UTC m=+2983.472959410" watchObservedRunningTime="2026-10-30 18:02:57.716218569 +0000 UTC m=+2983.478262386"
Oct 30 18:02:57.783887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.783626    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr" podStartSLOduration=2.783604335 podStartE2EDuration="2.783604335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:57.780733235 +0000 UTC m=+2983.542777049" watchObservedRunningTime="2026-10-30 18:02:57.783604335 +0000 UTC m=+2983.545648150"
Oct 30 18:02:57.883751 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.883716    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:57.944907 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.944818    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2\" (UniqueName: \"kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2\") pod \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") "
Oct 30 18:02:57.945090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.945029    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca\") pod \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") "
Oct 30 18:02:57.945158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.945132    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca\") pod \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\" (UID: \"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec\") "
Oct 30 18:02:57.945760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.945731    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca" pod "8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" (UID: "8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:02:57.946033 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.946009    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:57.949224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.949199    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2" pod "8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" (UID: "8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec"). InnerVolumeSpecName "kube-api-access-jb6k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:02:57.951844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:57.951817    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca" pod "8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" (UID: "8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:02:58.048052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:58.048022    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:58.048052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:58.048049    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb6k2\" (UniqueName: \"kubernetes.io/projected/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec-kube-api-access-jb6k2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:02:58.508939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:58.508905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:02:58.680816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:58.680727    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"
Oct 30 18:02:58.680816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:58.680762    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x" event={"ID":"8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec","Type":"ContainerDied","Data":"425b821091a046dff80e6f4b705feaf4eaa0b0f983f0f2bf7d1ccdecead649fc"}
Oct 30 18:02:58.680816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:02:58.680813    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="425b821091a046dff80e6f4b705feaf4eaa0b0f983f0f2bf7d1ccdecead649fc"
Oct 30 18:03:00.135538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:00.135500    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:01.899484 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:01.899450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:03.705608 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:03.705570    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:05.595833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:05.595796    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:07.003695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.001366    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:07.179556 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.179523    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"]
Oct 30 18:03:07.179778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.179754    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn" podUID="46c7f2b7-9a2a-4b50-8646-12ed8f29945c" containerName="maas-api" containerID="cri-o://6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b" gracePeriod=30
Oct 30 18:03:07.237814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.237778    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:07.356703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.356669    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"]
Oct 30 18:03:07.356918 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.356893    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing" containerID="cri-o://3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172" gracePeriod=30
Oct 30 18:03:07.457693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.456772    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:07.528686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.527348    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 18:03:07.549666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.548714    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls\") pod \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") "
Oct 30 18:03:07.549666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.548853    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k\" (UniqueName: \"kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k\") pod \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") "
Oct 30 18:03:07.549666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.548906    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca\") pod \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") "
Oct 30 18:03:07.549666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.549010    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs\") pod \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\" (UID: \"46c7f2b7-9a2a-4b50-8646-12ed8f29945c\") "
Oct 30 18:03:07.556750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.551690    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k" pod "46c7f2b7-9a2a-4b50-8646-12ed8f29945c" (UID: "46c7f2b7-9a2a-4b50-8646-12ed8f29945c"). InnerVolumeSpecName "kube-api-access-7gs6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:07.556750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.552157    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca" pod "46c7f2b7-9a2a-4b50-8646-12ed8f29945c" (UID: "46c7f2b7-9a2a-4b50-8646-12ed8f29945c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:07.561175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.561082    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs" pod "46c7f2b7-9a2a-4b50-8646-12ed8f29945c" (UID: "46c7f2b7-9a2a-4b50-8646-12ed8f29945c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:07.561175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.561151    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls" pod "46c7f2b7-9a2a-4b50-8646-12ed8f29945c" (UID: "46c7f2b7-9a2a-4b50-8646-12ed8f29945c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:07.578180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.577993    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:07.652383 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.652351    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:07.652383 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.652383    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:07.652744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.652396    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gs6k\" (UniqueName: \"kubernetes.io/projected/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-kube-api-access-7gs6k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:07.652744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.652405    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c7f2b7-9a2a-4b50-8646-12ed8f29945c-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:07.711676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.709275    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 18:03:07.884665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.884575    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume\") pod \"a017e36e-4f65-4737-a6e0-965ef6370a1d\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") "
Oct 30 18:03:07.884902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.884690    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p\" (UniqueName: \"kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p\") pod \"a017e36e-4f65-4737-a6e0-965ef6370a1d\" (UID: \"a017e36e-4f65-4737-a6e0-965ef6370a1d\") "
Oct 30 18:03:07.896665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.892004    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume" pod "a017e36e-4f65-4737-a6e0-965ef6370a1d" (UID: "a017e36e-4f65-4737-a6e0-965ef6370a1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:07.900441 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.900411    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p" pod "a017e36e-4f65-4737-a6e0-965ef6370a1d" (UID: "a017e36e-4f65-4737-a6e0-965ef6370a1d"). InnerVolumeSpecName "kube-api-access-4ms5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:07.913129 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.912872    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:07.914468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.914365    5534 generic.go:356] "Generic (PLEG): container finished" podID="46c7f2b7-9a2a-4b50-8646-12ed8f29945c" containerID="6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b" exitCode=0
Oct 30 18:03:07.914665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.914522    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn" event={"ID":"46c7f2b7-9a2a-4b50-8646-12ed8f29945c","Type":"ContainerDied","Data":"6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b"}
Oct 30 18:03:07.914665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.914571    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn" event={"ID":"46c7f2b7-9a2a-4b50-8646-12ed8f29945c","Type":"ContainerDied","Data":"e75aeadfc70641c2fafe0ed92656ef4c6f7efe42a0a965746f55a89f90b33f63"}
Oct 30 18:03:07.914665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.914601    5534 scope.go:122] "RemoveContainer" containerID="6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b"
Oct 30 18:03:07.914913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.914806    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"
Oct 30 18:03:07.944043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.943963    5534 generic.go:356] "Generic (PLEG): container finished" podID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerID="3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172" exitCode=0
Oct 30 18:03:07.944043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.944009    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" event={"ID":"a017e36e-4f65-4737-a6e0-965ef6370a1d","Type":"ContainerDied","Data":"3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172"}
Oct 30 18:03:07.945620 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.944187    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"
Oct 30 18:03:07.945932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.945834    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t" event={"ID":"a017e36e-4f65-4737-a6e0-965ef6370a1d","Type":"ContainerDied","Data":"62934a7761ae5d92c8ce07350b883a93efe7f83c06a5ae623b9462f67b92b8a5"}
Oct 30 18:03:07.958355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.958335    5534 scope.go:122] "RemoveContainer" containerID="6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b"
Oct 30 18:03:07.958695 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:07.958635    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b\": container with ID starting with 6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b not found: ID does not exist" containerID="6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b"
Oct 30 18:03:07.958872 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.958845    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b"} err="failed to get container status \"6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b\": rpc error: code = NotFound desc = could not find container \"6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b\": container with ID starting with 6006b948ebd9b3b95fd6a903eef5a30990aae99dee9a439bc19132ccd2dd1d8b not found: ID does not exist"
Oct 30 18:03:07.959005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.958994    5534 scope.go:122] "RemoveContainer" containerID="3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172"
Oct 30 18:03:07.974041 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.974002    5534 scope.go:122] "RemoveContainer" containerID="dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"
Oct 30 18:03:07.986993 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.986965    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a017e36e-4f65-4737-a6e0-965ef6370a1d-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:07.986993 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.986994    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ms5p\" (UniqueName: \"kubernetes.io/projected/a017e36e-4f65-4737-a6e0-965ef6370a1d-kube-api-access-4ms5p\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:07.987154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.987049    5534 scope.go:122] "RemoveContainer" containerID="3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172"
Oct 30 18:03:07.987382 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:07.987358    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172\": container with ID starting with 3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172 not found: ID does not exist" containerID="3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172"
Oct 30 18:03:07.987490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.987387    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172"} err="failed to get container status \"3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172\": rpc error: code = NotFound desc = could not find container \"3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172\": container with ID starting with 3150f289d7047ac94cda4fa20808e89746fdb52264b9f34e3677b32a3127a172 not found: ID does not exist"
Oct 30 18:03:07.987490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.987404    5534 scope.go:122] "RemoveContainer" containerID="dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"
Oct 30 18:03:07.987866 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:07.987839    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09\": container with ID starting with dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09 not found: ID does not exist" containerID="dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"
Oct 30 18:03:07.987969 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.987869    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09"} err="failed to get container status \"dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09\": rpc error: code = NotFound desc = could not find container \"dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09\": container with ID starting with dea16b8443455d8f99b66f90f43e3c6b2fb14ab88f7c1218b2ca51d292a38f09 not found: ID does not exist"
Oct 30 18:03:07.998304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:07.998282    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"]
Oct 30 18:03:08.005552 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.005528    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-21870f-66b759459f-fhrvn"]
Oct 30 18:03:08.053169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.053140    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"]
Oct 30 18:03:08.057281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.057260    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-8489fc575d-tcm4t"]
Oct 30 18:03:08.073892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.073869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:08.100434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.100409    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"]
Oct 30 18:03:08.104015 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.103997    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-21870f-nhq9c"]
Oct 30 18:03:08.243855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.243820    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:08.373076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.373044    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c7f2b7-9a2a-4b50-8646-12ed8f29945c" path="/var/lib/kubelet/pods/46c7f2b7-9a2a-4b50-8646-12ed8f29945c/volumes"
Oct 30 18:03:08.373547 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.373530    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" path="/var/lib/kubelet/pods/a017e36e-4f65-4737-a6e0-965ef6370a1d/volumes"
Oct 30 18:03:08.374084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.374050    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14ace23-27ce-4616-978e-73c6f214aa97" path="/var/lib/kubelet/pods/d14ace23-27ce-4616-978e-73c6f214aa97/volumes"
Oct 30 18:03:08.582895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.582819    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:08.683753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.683727    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:03:08.683922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:08.683792    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:03:09.711967 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:09.711929    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:09.923664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:09.908880    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:10.312669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.312219    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"]
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323436    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323463    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323480    5534 cpu_manager.go:414] "containerMap: removing container" podUID="46c7f2b7-9a2a-4b50-8646-12ed8f29945c" containerName="maas-api"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323488    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c7f2b7-9a2a-4b50-8646-12ed8f29945c" containerName="maas-api"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323534    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" containerName="revoke-keys"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323542    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" containerName="revoke-keys"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323725    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c7f2b7-9a2a-4b50-8646-12ed8f29945c" containerName="maas-api"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323743    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing"
Oct 30 18:03:10.331666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.323758    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" containerName="revoke-keys"
Oct 30 18:03:10.363669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.362990    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.428682 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.395991    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5cbb4\""
Oct 30 18:03:10.428682 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.426898    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.428682 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.427033    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl92h\" (UniqueName: \"kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h\") pod \"payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.517326 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.495982    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b5cbb4-dockercfg-4nf7q\""
Oct 30 18:03:10.551704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.537200    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.551704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.537373    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl92h\" (UniqueName: \"kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h\") pod \"payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.551704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.538719    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.562372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.562343    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"]
Oct 30 18:03:10.565549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.565477    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl92h\" (UniqueName: \"kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h\") pod \"payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.825272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.825195    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:10.937597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:10.937566    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:11.142099 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:03:11.142067    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70c6ad7_716e_4fd0_8c28_eef78f8543e8.slice/crio-c9bacbde3e4d14345737988200b1718354499c602b603f51e2a8f1158b49ddae WatchSource:0}: Error finding container c9bacbde3e4d14345737988200b1718354499c602b603f51e2a8f1158b49ddae: Status 404 returned error can't find the container with id c9bacbde3e4d14345737988200b1718354499c602b603f51e2a8f1158b49ddae
Oct 30 18:03:11.142340 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:11.142317    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"]
Oct 30 18:03:11.589412 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:11.589316    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:11.999818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:11.999780    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt" event={"ID":"f70c6ad7-716e-4fd0-8c28-eef78f8543e8","Type":"ContainerStarted","Data":"e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785"}
Oct 30 18:03:12.000246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:11.999849    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt" event={"ID":"f70c6ad7-716e-4fd0-8c28-eef78f8543e8","Type":"ContainerStarted","Data":"c9bacbde3e4d14345737988200b1718354499c602b603f51e2a8f1158b49ddae"}
Oct 30 18:03:12.000246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:11.999880    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:12.058213 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:12.058147    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt" podStartSLOduration=2.058131607 podStartE2EDuration="2.058131607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:12.048809052 +0000 UTC m=+2997.810852872" watchObservedRunningTime="2026-10-30 18:03:12.058131607 +0000 UTC m=+2997.820175444"
Oct 30 18:03:12.104593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:12.104559    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:12.243666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:12.243506    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:12.795401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:12.795369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:14.743869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:14.743835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:14.881535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:14.881497    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:14.998390 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:14.998285    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:15.193573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.193535    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"]
Oct 30 18:03:15.193826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.193791    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" containerID="cri-o://7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708" gracePeriod=30
Oct 30 18:03:15.248977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.248885    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:15.253816 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:15.253780    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:03:15.271576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.271553    5534 patch_prober.go:29] interesting pod/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:03:15.271735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.271601    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:03:15.462482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.461382    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:15.777133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.777103    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"]
Oct 30 18:03:15.777914 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.777897    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing"
Oct 30 18:03:15.777991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.777915    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing"
Oct 30 18:03:15.778090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.778081    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a017e36e-4f65-4737-a6e0-965ef6370a1d" containerName="payload-processing"
Oct 30 18:03:15.781780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.781761    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.797145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.797126    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:03:15.797279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.797268    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:03:15.802222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.802204    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"]
Oct 30 18:03:15.890502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.890468    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.890502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.890516    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.890904 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.890876    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwlv\" (UniqueName: \"kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.966102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.966072    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:15.993038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.993008    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.993246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.993052    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.993346 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.993324    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwlv\" (UniqueName: \"kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.993778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.993747    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:15.995768 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:15.995737    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:16.025955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.025929    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwlv\" (UniqueName: \"kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv\") pod \"maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:16.096721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.096694    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:16.105971 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.105937    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:16.335463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.335435    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:16.427449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.427409    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"]
Oct 30 18:03:16.428117 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:03:16.428090    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9b530a_0fac_4fbe_a34b_df2535a8d103.slice/crio-f67b7353edffafe31071abcaf45bc028d105c528c5a8c5a352dc07d8b637988f WatchSource:0}: Error finding container f67b7353edffafe31071abcaf45bc028d105c528c5a8c5a352dc07d8b637988f: Status 404 returned error can't find the container with id f67b7353edffafe31071abcaf45bc028d105c528c5a8c5a352dc07d8b637988f
Oct 30 18:03:16.870097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.870062    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"]
Oct 30 18:03:16.909621 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:16.909584    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"]
Oct 30 18:03:17.025738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.025625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:17.051370 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.051332    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln" event={"ID":"5c9b530a-0fac-4fbe-a34b-df2535a8d103","Type":"ContainerStarted","Data":"1d0911f407008602c0e71393c4a3384fc1e84243d6680b4653d712ba8d5a4117"}
Oct 30 18:03:17.051553 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.051391    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln" event={"ID":"5c9b530a-0fac-4fbe-a34b-df2535a8d103","Type":"ContainerStarted","Data":"f67b7353edffafe31071abcaf45bc028d105c528c5a8c5a352dc07d8b637988f"}
Oct 30 18:03:17.135585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.134571    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln" podStartSLOduration=2.134555573 podStartE2EDuration="2.134555573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:17.128788002 +0000 UTC m=+3002.890831816" watchObservedRunningTime="2026-10-30 18:03:17.134555573 +0000 UTC m=+3002.896599387"
Oct 30 18:03:17.172610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.172579    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:17.364826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.364798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:17.506343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:17.506315    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:18.014905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.014874    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"]
Oct 30 18:03:18.019862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.019843    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.046337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.046315    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"]
Oct 30 18:03:18.069155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.069118    5534 generic.go:356] "Generic (PLEG): container finished" podID="5c9b530a-0fac-4fbe-a34b-df2535a8d103" containerID="1d0911f407008602c0e71393c4a3384fc1e84243d6680b4653d712ba8d5a4117" exitCode=0
Oct 30 18:03:18.069346 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.069245    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln" event={"ID":"5c9b530a-0fac-4fbe-a34b-df2535a8d103","Type":"ContainerDied","Data":"1d0911f407008602c0e71393c4a3384fc1e84243d6680b4653d712ba8d5a4117"}
Oct 30 18:03:18.145346 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.145309    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.145518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.145450    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2h6\" (UniqueName: \"kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.145717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.145693    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.247396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.247363    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.247561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.247492    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.247561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.247554    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2h6\" (UniqueName: \"kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.248251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.248227    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.249970 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.249948    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.270677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.270592    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2h6\" (UniqueName: \"kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.334343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.334303    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:18.647011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:18.646982    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"]
Oct 30 18:03:19.079267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.079233    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6" event={"ID":"367a0e22-06df-4ede-b159-0636e793712d","Type":"ContainerStarted","Data":"aa58df6e2d5e4d4d744d350b1780a4e876b141f7524fdee7648250a6bf4785b2"}
Oct 30 18:03:19.079782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.079297    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6" event={"ID":"367a0e22-06df-4ede-b159-0636e793712d","Type":"ContainerStarted","Data":"7e09e6312d3782356b8f5319634d8097c94179a5d71a6416afdffc28c56be3de"}
Oct 30 18:03:19.123720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.123634    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6" podStartSLOduration=2.123614074 podStartE2EDuration="2.123614074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:19.120131435 +0000 UTC m=+3004.882175240" watchObservedRunningTime="2026-10-30 18:03:19.123614074 +0000 UTC m=+3004.885657889"
Oct 30 18:03:19.238385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.238357    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:19.368765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.368732    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca\") pod \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") "
Oct 30 18:03:19.368975 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.368815    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv\" (UniqueName: \"kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv\") pod \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") "
Oct 30 18:03:19.368975 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.368845    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca\") pod \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\" (UID: \"5c9b530a-0fac-4fbe-a34b-df2535a8d103\") "
Oct 30 18:03:19.369150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.369122    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca" pod "5c9b530a-0fac-4fbe-a34b-df2535a8d103" (UID: "5c9b530a-0fac-4fbe-a34b-df2535a8d103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:19.369386 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.369363    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:19.371327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.371300    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca" pod "5c9b530a-0fac-4fbe-a34b-df2535a8d103" (UID: "5c9b530a-0fac-4fbe-a34b-df2535a8d103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:19.371490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.371466    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv" pod "5c9b530a-0fac-4fbe-a34b-df2535a8d103" (UID: "5c9b530a-0fac-4fbe-a34b-df2535a8d103"). InnerVolumeSpecName "kube-api-access-6nwlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:19.448111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.448079    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:19.470763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.470725    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nwlv\" (UniqueName: \"kubernetes.io/projected/5c9b530a-0fac-4fbe-a34b-df2535a8d103-kube-api-access-6nwlv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:19.470763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.470753    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9b530a-0fac-4fbe-a34b-df2535a8d103-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:19.550538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.550500    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"]
Oct 30 18:03:19.551399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.551378    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5c9b530a-0fac-4fbe-a34b-df2535a8d103" containerName="revoke-keys"
Oct 30 18:03:19.551399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.551398    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9b530a-0fac-4fbe-a34b-df2535a8d103" containerName="revoke-keys"
Oct 30 18:03:19.551613 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.551600    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9b530a-0fac-4fbe-a34b-df2535a8d103" containerName="revoke-keys"
Oct 30 18:03:19.557980 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.557962    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.573290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.573263    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96c514\""
Oct 30 18:03:19.573405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.573321    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96c514-dockercfg-xtg9l\""
Oct 30 18:03:19.576835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.576813    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"]
Oct 30 18:03:19.591799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.591774    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:19.619835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.619747    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"]
Oct 30 18:03:19.631819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.631791    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.648047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.648007    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"]
Oct 30 18:03:19.679300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.679261    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.679478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.679326    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwtr\" (UniqueName: \"kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr\") pod \"payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.713198 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.713095    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:19.837404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.837375    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.838049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.837996    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4w8\" (UniqueName: \"kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8\") pod \"payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.838529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.838508    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.839534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.839516    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwtr\" (UniqueName: \"kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr\") pod \"payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.840204 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.839444    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.957709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.953486    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4w8\" (UniqueName: \"kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8\") pod \"payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.957709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.953734    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.957709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.954951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.959726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.959561    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwtr\" (UniqueName: \"kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr\") pod \"payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:19.978760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.978728    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4w8\" (UniqueName: \"kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8\") pod \"payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:19.991780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:19.991754    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:20.042606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.042490    5534 scope.go:122] "RemoveContainer" containerID="677a87b179687150a19e59fb157a812ec786753199469365ed7db441e1d20b5b"
Oct 30 18:03:20.097958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.097928    5534 generic.go:356] "Generic (PLEG): container finished" podID="367a0e22-06df-4ede-b159-0636e793712d" containerID="aa58df6e2d5e4d4d744d350b1780a4e876b141f7524fdee7648250a6bf4785b2" exitCode=0
Oct 30 18:03:20.098389 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.098017    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6" event={"ID":"367a0e22-06df-4ede-b159-0636e793712d","Type":"ContainerDied","Data":"aa58df6e2d5e4d4d744d350b1780a4e876b141f7524fdee7648250a6bf4785b2"}
Oct 30 18:03:20.100768 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.100752    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"
Oct 30 18:03:20.100865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.100764    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln" event={"ID":"5c9b530a-0fac-4fbe-a34b-df2535a8d103","Type":"ContainerDied","Data":"f67b7353edffafe31071abcaf45bc028d105c528c5a8c5a352dc07d8b637988f"}
Oct 30 18:03:20.100865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.100807    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f67b7353edffafe31071abcaf45bc028d105c528c5a8c5a352dc07d8b637988f"
Oct 30 18:03:20.179469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.174177    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:20.318103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.316127    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"]
Oct 30 18:03:20.318103 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:03:20.316440    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491f8f7_bf5f_4240_b398_4d659c16ed39.slice/crio-39eb335588bfb6c6e911229a4b937355ddcf37395d7b768894ecdc7f7e886312 WatchSource:0}: Error finding container 39eb335588bfb6c6e911229a4b937355ddcf37395d7b768894ecdc7f7e886312: Status 404 returned error can't find the container with id 39eb335588bfb6c6e911229a4b937355ddcf37395d7b768894ecdc7f7e886312
Oct 30 18:03:20.498874 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.497773    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"]
Oct 30 18:03:20.546533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.546510    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 18:03:20.684453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684421    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684461    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684503    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684552    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684609    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684737    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684796    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.684948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684891    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:03:20.684948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684927    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.685169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684985    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data\") pod \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\" (UID: \"3612c751-1efc-4498-b1aa-2b8e47e45e8c\") "
Oct 30 18:03:20.685169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.684999    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:03:20.685169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685153    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:03:20.685512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685352    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:20.685512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685425    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:03:20.685811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685790    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.685927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685815    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.685927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685830    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.685927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685843    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.685927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.685855    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.687252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.687230    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "kube-api-access-zk4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:20.687984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.687961    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:20.688106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.687982    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:03:20.688106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.688071    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo" pod "3612c751-1efc-4498-b1aa-2b8e47e45e8c" (UID: "3612c751-1efc-4498-b1aa-2b8e47e45e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:03:20.787664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.787570    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4p6\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-kube-api-access-zk4p6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.787664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.787596    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.787664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.787605    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:20.787664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:20.787614    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3612c751-1efc-4498-b1aa-2b8e47e45e8c-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:21.111043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.110993    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g" event={"ID":"a491f8f7-bf5f-4240-b398-4d659c16ed39","Type":"ContainerStarted","Data":"a1b2cd035c3f0ffa19bf936511d59865ae9198fc8ac36796ffd2638acde2dadc"}
Oct 30 18:03:21.111593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.111061    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g" event={"ID":"a491f8f7-bf5f-4240-b398-4d659c16ed39","Type":"ContainerStarted","Data":"39eb335588bfb6c6e911229a4b937355ddcf37395d7b768894ecdc7f7e886312"}
Oct 30 18:03:21.111593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.111212    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:21.113002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.112972    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc" event={"ID":"9001bbcd-9da7-4b56-99aa-a255c145ea41","Type":"ContainerStarted","Data":"f81c1db03d3ad148bc0c4cfcc810976dcdfbf4eee2fee7803b08666c289a1b61"}
Oct 30 18:03:21.113131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.113033    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:21.113131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.113058    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc" event={"ID":"9001bbcd-9da7-4b56-99aa-a255c145ea41","Type":"ContainerStarted","Data":"1a639270ce81c534611fd998c778cf3af78c80a4d09aaefcd84f5816e435eacf"}
Oct 30 18:03:21.114720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.114698    5534 generic.go:356] "Generic (PLEG): container finished" podID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerID="7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708" exitCode=0
Oct 30 18:03:21.114829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.114740    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"
Oct 30 18:03:21.114829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.114765    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" event={"ID":"3612c751-1efc-4498-b1aa-2b8e47e45e8c","Type":"ContainerDied","Data":"7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708"}
Oct 30 18:03:21.114829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.114798    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml" event={"ID":"3612c751-1efc-4498-b1aa-2b8e47e45e8c","Type":"ContainerDied","Data":"9b97d6f02a58c9efaf6e11154ccbb6ed23855a02a45e8b57d9ee35dc66e06612"}
Oct 30 18:03:21.114829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.114820    5534 scope.go:122] "RemoveContainer" containerID="7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708"
Oct 30 18:03:21.128897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.128869    5534 scope.go:122] "RemoveContainer" containerID="e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"
Oct 30 18:03:21.143304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.143275    5534 scope.go:122] "RemoveContainer" containerID="7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708"
Oct 30 18:03:21.143777 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:21.143755    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708\": container with ID starting with 7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708 not found: ID does not exist" containerID="7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708"
Oct 30 18:03:21.143855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.143789    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708"} err="failed to get container status \"7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708\": rpc error: code = NotFound desc = could not find container \"7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708\": container with ID starting with 7a3feb9afce16bf64467b41517bb6dfa730cd205ab1a089520e75540b8e61708 not found: ID does not exist"
Oct 30 18:03:21.143855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.143809    5534 scope.go:122] "RemoveContainer" containerID="e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"
Oct 30 18:03:21.144138 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:21.144107    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb\": container with ID starting with e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb not found: ID does not exist" containerID="e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"
Oct 30 18:03:21.144138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.144130    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb"} err="failed to get container status \"e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb\": rpc error: code = NotFound desc = could not find container \"e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb\": container with ID starting with e5dc5e0c3309388acff993b4afbf28120e1b0b56664c2f6d75a8702cc34a75cb not found: ID does not exist"
Oct 30 18:03:21.207943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.207879    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g" podStartSLOduration=2.207861997 podStartE2EDuration="2.207861997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:21.207101092 +0000 UTC m=+3006.969144908" watchObservedRunningTime="2026-10-30 18:03:21.207861997 +0000 UTC m=+3006.969905811"
Oct 30 18:03:21.283193 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.283071    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"]
Oct 30 18:03:21.291631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.291597    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-21870f-openshift-default-6444b9b87-5cmml"]
Oct 30 18:03:21.335864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.335816    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc" podStartSLOduration=2.335799671 podStartE2EDuration="2.335799671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:21.332112789 +0000 UTC m=+3007.094156597" watchObservedRunningTime="2026-10-30 18:03:21.335799671 +0000 UTC m=+3007.097843486"
Oct 30 18:03:21.364294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.364202    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:21.504634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.504600    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6\" (UniqueName: \"kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6\") pod \"367a0e22-06df-4ede-b159-0636e793712d\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") "
Oct 30 18:03:21.504634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.504707    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca\") pod \"367a0e22-06df-4ede-b159-0636e793712d\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") "
Oct 30 18:03:21.505043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.505019    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca\") pod \"367a0e22-06df-4ede-b159-0636e793712d\" (UID: \"367a0e22-06df-4ede-b159-0636e793712d\") "
Oct 30 18:03:21.508992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.508965    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca" pod "367a0e22-06df-4ede-b159-0636e793712d" (UID: "367a0e22-06df-4ede-b159-0636e793712d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:21.511745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.509594    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6" pod "367a0e22-06df-4ede-b159-0636e793712d" (UID: "367a0e22-06df-4ede-b159-0636e793712d"). InnerVolumeSpecName "kube-api-access-7f2h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:21.511745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.510054    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca" pod "367a0e22-06df-4ede-b159-0636e793712d" (UID: "367a0e22-06df-4ede-b159-0636e793712d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:21.519672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.519552    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:21.608764 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.608727    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367a0e22-06df-4ede-b159-0636e793712d-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:21.608764 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.608760    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2h6\" (UniqueName: \"kubernetes.io/projected/367a0e22-06df-4ede-b159-0636e793712d-kube-api-access-7f2h6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:21.608994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.608775    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367a0e22-06df-4ede-b159-0636e793712d-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:21.870937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.870889    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg" podUID="cee5267d-e499-4cad-8e7f-ab4080952027" containerName="payload-processing" containerID="cri-o://11fca12c12e04c8c5f708cc6e6f2886c849d6d50f7cad46064896dd49fe5d283" gracePeriod=25
Oct 30 18:03:21.909913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:21.909855    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns" podUID="741e5fb2-53df-4dd8-8528-1f1f0c10afe9" containerName="payload-pre-processing" containerID="cri-o://91a82b89660f44ac0338402617445ed359b1609b7d8a047168ea92d05c012b20" gracePeriod=25
Oct 30 18:03:22.131751 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.131661    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6" event={"ID":"367a0e22-06df-4ede-b159-0636e793712d","Type":"ContainerDied","Data":"7e09e6312d3782356b8f5319634d8097c94179a5d71a6416afdffc28c56be3de"}
Oct 30 18:03:22.132232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.131764    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e09e6312d3782356b8f5319634d8097c94179a5d71a6416afdffc28c56be3de"
Oct 30 18:03:22.132232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.131849    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"
Oct 30 18:03:22.135950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.135741    5534 generic.go:356] "Generic (PLEG): container finished" podID="cee5267d-e499-4cad-8e7f-ab4080952027" containerID="11fca12c12e04c8c5f708cc6e6f2886c849d6d50f7cad46064896dd49fe5d283" exitCode=0
Oct 30 18:03:22.135950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.135869    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg" event={"ID":"cee5267d-e499-4cad-8e7f-ab4080952027","Type":"ContainerDied","Data":"11fca12c12e04c8c5f708cc6e6f2886c849d6d50f7cad46064896dd49fe5d283"}
Oct 30 18:03:22.138726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.138177    5534 generic.go:356] "Generic (PLEG): container finished" podID="741e5fb2-53df-4dd8-8528-1f1f0c10afe9" containerID="91a82b89660f44ac0338402617445ed359b1609b7d8a047168ea92d05c012b20" exitCode=0
Oct 30 18:03:22.139222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.139199    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns" event={"ID":"741e5fb2-53df-4dd8-8528-1f1f0c10afe9","Type":"ContainerDied","Data":"91a82b89660f44ac0338402617445ed359b1609b7d8a047168ea92d05c012b20"}
Oct 30 18:03:22.197172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.196943    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:03:22.308814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.308788    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:03:22.319635 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.319606    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6\" (UniqueName: \"kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6\") pod \"cee5267d-e499-4cad-8e7f-ab4080952027\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") "
Oct 30 18:03:22.319852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.319832    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume\") pod \"cee5267d-e499-4cad-8e7f-ab4080952027\" (UID: \"cee5267d-e499-4cad-8e7f-ab4080952027\") "
Oct 30 18:03:22.320172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.320143    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume" pod "cee5267d-e499-4cad-8e7f-ab4080952027" (UID: "cee5267d-e499-4cad-8e7f-ab4080952027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:22.320499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.320481    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5267d-e499-4cad-8e7f-ab4080952027-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:22.322011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.321984    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6" pod "cee5267d-e499-4cad-8e7f-ab4080952027" (UID: "cee5267d-e499-4cad-8e7f-ab4080952027"). InnerVolumeSpecName "kube-api-access-cp7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:22.379628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.379587    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" path="/var/lib/kubelet/pods/3612c751-1efc-4498-b1aa-2b8e47e45e8c/volumes"
Oct 30 18:03:22.421532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.421500    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume\") pod \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") "
Oct 30 18:03:22.421721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.421678    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq\" (UniqueName: \"kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq\") pod \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\" (UID: \"741e5fb2-53df-4dd8-8528-1f1f0c10afe9\") "
Oct 30 18:03:22.421957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.421932    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume" pod "741e5fb2-53df-4dd8-8528-1f1f0c10afe9" (UID: "741e5fb2-53df-4dd8-8528-1f1f0c10afe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:22.422287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.422270    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:22.422346 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.422289    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp7z6\" (UniqueName: \"kubernetes.io/projected/cee5267d-e499-4cad-8e7f-ab4080952027-kube-api-access-cp7z6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:22.423979 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.423951    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq" pod "741e5fb2-53df-4dd8-8528-1f1f0c10afe9" (UID: "741e5fb2-53df-4dd8-8528-1f1f0c10afe9"). InnerVolumeSpecName "kube-api-access-phxpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:22.524051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:22.524021    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxpq\" (UniqueName: \"kubernetes.io/projected/741e5fb2-53df-4dd8-8528-1f1f0c10afe9-kube-api-access-phxpq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:23.017283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.017254    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:03:23.148096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.148066    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"
Oct 30 18:03:23.148096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.148086    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg" event={"ID":"cee5267d-e499-4cad-8e7f-ab4080952027","Type":"ContainerDied","Data":"a84389f9c475e82f4e4048cbbe60e7ad14592a830e2d2889726a21d62b1dfc8c"}
Oct 30 18:03:23.148592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.148144    5534 scope.go:122] "RemoveContainer" containerID="11fca12c12e04c8c5f708cc6e6f2886c849d6d50f7cad46064896dd49fe5d283"
Oct 30 18:03:23.150829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.150060    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns" event={"ID":"741e5fb2-53df-4dd8-8528-1f1f0c10afe9","Type":"ContainerDied","Data":"7db7b0da35b6b1301974b1d24679a84e4a024fdb816ebce479d9de0320e0e127"}
Oct 30 18:03:23.150829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.150070    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"
Oct 30 18:03:23.162103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.162084    5534 scope.go:122] "RemoveContainer" containerID="91a82b89660f44ac0338402617445ed359b1609b7d8a047168ea92d05c012b20"
Oct 30 18:03:23.202501 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.202472    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"]
Oct 30 18:03:23.209625 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.209597    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-55d6b57b7d-vtkrg"]
Oct 30 18:03:23.259261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.259230    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"]
Oct 30 18:03:23.267000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.266972    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-647c7cc47f-6rzns"]
Oct 30 18:03:23.316805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:23.316725    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:24.374957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:24.374924    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741e5fb2-53df-4dd8-8528-1f1f0c10afe9" path="/var/lib/kubelet/pods/741e5fb2-53df-4dd8-8528-1f1f0c10afe9/volumes"
Oct 30 18:03:24.375446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:24.375356    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee5267d-e499-4cad-8e7f-ab4080952027" path="/var/lib/kubelet/pods/cee5267d-e499-4cad-8e7f-ab4080952027/volumes"
Oct 30 18:03:25.006253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:25.006221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:26.708199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:26.708160    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:27.172779 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:27.172745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:27.504823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:27.504734    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:28.566378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:28.566348    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:28.686796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:28.686762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:28.860085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:28.860055    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:29.348341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:29.348309    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:29.474256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:29.474224    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:30.509176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:30.509138    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:32.142688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:32.142660    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:03:32.143081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:32.142727    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:03:32.200766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:32.200727    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:33.915878 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:33.915846    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:35.032049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.032014    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"]
Oct 30 18:03:35.033037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033013    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy"
Oct 30 18:03:35.033037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033038    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033065    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033072    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033109    5534 cpu_manager.go:414] "containerMap: removing container" podUID="741e5fb2-53df-4dd8-8528-1f1f0c10afe9" containerName="payload-pre-processing"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033117    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741e5fb2-53df-4dd8-8528-1f1f0c10afe9" containerName="payload-pre-processing"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033149    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cee5267d-e499-4cad-8e7f-ab4080952027" containerName="payload-processing"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033156    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee5267d-e499-4cad-8e7f-ab4080952027" containerName="payload-processing"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033184    5534 cpu_manager.go:414] "containerMap: removing container" podUID="367a0e22-06df-4ede-b159-0636e793712d" containerName="revoke-keys"
Oct 30 18:03:35.033231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033193    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367a0e22-06df-4ede-b159-0636e793712d" containerName="revoke-keys"
Oct 30 18:03:35.033636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033376    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy"
Oct 30 18:03:35.033636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033397    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="741e5fb2-53df-4dd8-8528-1f1f0c10afe9" containerName="payload-pre-processing"
Oct 30 18:03:35.033636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033412    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee5267d-e499-4cad-8e7f-ab4080952027" containerName="payload-processing"
Oct 30 18:03:35.033636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.033428    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="367a0e22-06df-4ede-b159-0636e793712d" containerName="revoke-keys"
Oct 30 18:03:35.039337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.039314    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.058054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.058029    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"]
Oct 30 18:03:35.064355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.064330    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:03:35.064579 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.064565    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:03:35.168724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.168669    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.168940 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.168803    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.168940 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.168913    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsb8s\" (UniqueName: \"kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.270726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.270682    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.270931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.270769    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.270931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.270839    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsb8s\" (UniqueName: \"kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.271343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.271322    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.273500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.273465    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.294175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.294112    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsb8s\" (UniqueName: \"kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s\") pod \"maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.354247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.354219    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:35.674392 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:03:35.674364    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d6e400_bca3_48cc_a421_8b356760cc5a.slice/crio-677db991d2ef7e0c481ded737aadfb240a29c26435fb6620f69a618a8a6321b4 WatchSource:0}: Error finding container 677db991d2ef7e0c481ded737aadfb240a29c26435fb6620f69a618a8a6321b4: Status 404 returned error can't find the container with id 677db991d2ef7e0c481ded737aadfb240a29c26435fb6620f69a618a8a6321b4
Oct 30 18:03:35.677065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.677036    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"]
Oct 30 18:03:35.847053 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:35.847019    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:36.282478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:36.282436    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5" event={"ID":"59d6e400-bca3-48cc-a421-8b356760cc5a","Type":"ContainerStarted","Data":"6ee4aff4e0907d9bddb162238072ca56a91d7d98833d98189f5a2cfa309593a5"}
Oct 30 18:03:36.283016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:36.282510    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5" event={"ID":"59d6e400-bca3-48cc-a421-8b356760cc5a","Type":"ContainerStarted","Data":"677db991d2ef7e0c481ded737aadfb240a29c26435fb6620f69a618a8a6321b4"}
Oct 30 18:03:36.324704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:36.324596    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5" podStartSLOduration=1.324576305 podStartE2EDuration="1.324576305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:36.321787772 +0000 UTC m=+3022.083831578" watchObservedRunningTime="2026-10-30 18:03:36.324576305 +0000 UTC m=+3022.086620120"
Oct 30 18:03:37.295334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:37.295299    5534 generic.go:356] "Generic (PLEG): container finished" podID="59d6e400-bca3-48cc-a421-8b356760cc5a" containerID="6ee4aff4e0907d9bddb162238072ca56a91d7d98833d98189f5a2cfa309593a5" exitCode=0
Oct 30 18:03:37.295864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:37.295393    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5" event={"ID":"59d6e400-bca3-48cc-a421-8b356760cc5a","Type":"ContainerDied","Data":"6ee4aff4e0907d9bddb162238072ca56a91d7d98833d98189f5a2cfa309593a5"}
Oct 30 18:03:38.853604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.853578    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:38.878157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.878111    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:38.967802 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.967706    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca\") pod \"59d6e400-bca3-48cc-a421-8b356760cc5a\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") "
Oct 30 18:03:38.967802 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.967773    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s\" (UniqueName: \"kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s\") pod \"59d6e400-bca3-48cc-a421-8b356760cc5a\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") "
Oct 30 18:03:38.968033 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.968000    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca\") pod \"59d6e400-bca3-48cc-a421-8b356760cc5a\" (UID: \"59d6e400-bca3-48cc-a421-8b356760cc5a\") "
Oct 30 18:03:38.971285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.971260    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca" pod "59d6e400-bca3-48cc-a421-8b356760cc5a" (UID: "59d6e400-bca3-48cc-a421-8b356760cc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:38.973506 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.973478    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s" pod "59d6e400-bca3-48cc-a421-8b356760cc5a" (UID: "59d6e400-bca3-48cc-a421-8b356760cc5a"). InnerVolumeSpecName "kube-api-access-gsb8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:38.973615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:38.973571    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca" pod "59d6e400-bca3-48cc-a421-8b356760cc5a" (UID: "59d6e400-bca3-48cc-a421-8b356760cc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:39.075516 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:39.070092    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:39.075516 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:39.070125    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsb8s\" (UniqueName: \"kubernetes.io/projected/59d6e400-bca3-48cc-a421-8b356760cc5a-kube-api-access-gsb8s\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:39.075516 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:39.070139    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d6e400-bca3-48cc-a421-8b356760cc5a-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:39.373938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:39.373914    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"
Oct 30 18:03:39.374121 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:39.373952    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5" event={"ID":"59d6e400-bca3-48cc-a421-8b356760cc5a","Type":"ContainerDied","Data":"677db991d2ef7e0c481ded737aadfb240a29c26435fb6620f69a618a8a6321b4"}
Oct 30 18:03:39.374121 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:39.373996    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="677db991d2ef7e0c481ded737aadfb240a29c26435fb6620f69a618a8a6321b4"
Oct 30 18:03:40.695996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:40.695961    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:42.308213 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:42.308177    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:44.209807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:44.209773    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:45.246852 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:03:45.246818    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:03:45.928625 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:45.928594    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:47.268002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.267963    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"]
Oct 30 18:03:47.268530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.268232    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9" podUID="d3e5f9bc-0833-4071-bb36-941e32e98430" containerName="maas-api" containerID="cri-o://426d272b5f757cc9a2f48e8a752d42125ebe6b17feeb6079cc1537f659a5f970" gracePeriod=30
Oct 30 18:03:47.347057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.347021    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:47.449187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.449151    5534 generic.go:356] "Generic (PLEG): container finished" podID="d3e5f9bc-0833-4071-bb36-941e32e98430" containerID="426d272b5f757cc9a2f48e8a752d42125ebe6b17feeb6079cc1537f659a5f970" exitCode=0
Oct 30 18:03:47.449348 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.449336    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9" event={"ID":"d3e5f9bc-0833-4071-bb36-941e32e98430","Type":"ContainerDied","Data":"426d272b5f757cc9a2f48e8a752d42125ebe6b17feeb6079cc1537f659a5f970"}
Oct 30 18:03:47.570737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.570559    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:47.609781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.609756    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:03:47.689291 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.689256    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:47.720532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.720481    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs\") pod \"d3e5f9bc-0833-4071-bb36-941e32e98430\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") "
Oct 30 18:03:47.720752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.720669    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls\") pod \"d3e5f9bc-0833-4071-bb36-941e32e98430\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") "
Oct 30 18:03:47.720959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.720939    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft\" (UniqueName: \"kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft\") pod \"d3e5f9bc-0833-4071-bb36-941e32e98430\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") "
Oct 30 18:03:47.721077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.721058    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca\") pod \"d3e5f9bc-0833-4071-bb36-941e32e98430\" (UID: \"d3e5f9bc-0833-4071-bb36-941e32e98430\") "
Oct 30 18:03:47.722393 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.722365    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca" pod "d3e5f9bc-0833-4071-bb36-941e32e98430" (UID: "d3e5f9bc-0833-4071-bb36-941e32e98430"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:47.730367 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.730331    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs" pod "d3e5f9bc-0833-4071-bb36-941e32e98430" (UID: "d3e5f9bc-0833-4071-bb36-941e32e98430"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:47.730734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.730703    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls" pod "d3e5f9bc-0833-4071-bb36-941e32e98430" (UID: "d3e5f9bc-0833-4071-bb36-941e32e98430"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:47.739368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.739317    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft" pod "d3e5f9bc-0833-4071-bb36-941e32e98430" (UID: "d3e5f9bc-0833-4071-bb36-941e32e98430"). InnerVolumeSpecName "kube-api-access-kg2ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:47.830086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.829919    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:47.830086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.829957    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2ft\" (UniqueName: \"kubernetes.io/projected/d3e5f9bc-0833-4071-bb36-941e32e98430-kube-api-access-kg2ft\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:47.830086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.829970    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e5f9bc-0833-4071-bb36-941e32e98430-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:47.830086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.829982    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f9bc-0833-4071-bb36-941e32e98430-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:47.933601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:47.933466    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:48.084922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.084835    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:48.120054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.120019    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"]
Oct 30 18:03:48.124441 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.124411    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e927d7-tzhln"]
Oct 30 18:03:48.202917 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.202883    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:48.371529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.371495    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9b530a-0fac-4fbe-a34b-df2535a8d103" path="/var/lib/kubelet/pods/5c9b530a-0fac-4fbe-a34b-df2535a8d103/volumes"
Oct 30 18:03:48.461235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.461100    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"
Oct 30 18:03:48.461235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.461135    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9" event={"ID":"d3e5f9bc-0833-4071-bb36-941e32e98430","Type":"ContainerDied","Data":"07b0654d9590ac545d6895e9345f693bf9979bb31963eb66cad7290386617f39"}
Oct 30 18:03:48.461235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.461200    5534 scope.go:122] "RemoveContainer" containerID="426d272b5f757cc9a2f48e8a752d42125ebe6b17feeb6079cc1537f659a5f970"
Oct 30 18:03:48.524349 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.524318    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"]
Oct 30 18:03:48.535852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.535829    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e927d7-67948cc4f8-rfdl9"]
Oct 30 18:03:48.622584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:48.621246    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:49.934344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:49.934307    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:50.374261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:50.374227    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e5f9bc-0833-4071-bb36-941e32e98430" path="/var/lib/kubelet/pods/d3e5f9bc-0833-4071-bb36-941e32e98430/volumes"
Oct 30 18:03:52.147094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:52.147062    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:53.550443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.550399    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"]
Oct 30 18:03:53.551263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551245    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d3e5f9bc-0833-4071-bb36-941e32e98430" containerName="maas-api"
Oct 30 18:03:53.551263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551263    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e5f9bc-0833-4071-bb36-941e32e98430" containerName="maas-api"
Oct 30 18:03:53.551399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551289    5534 cpu_manager.go:414] "containerMap: removing container" podUID="59d6e400-bca3-48cc-a421-8b356760cc5a" containerName="revoke-keys"
Oct 30 18:03:53.551399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551294    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d6e400-bca3-48cc-a421-8b356760cc5a" containerName="revoke-keys"
Oct 30 18:03:53.551518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551415    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3612c751-1efc-4498-b1aa-2b8e47e45e8c" containerName="istio-proxy"
Oct 30 18:03:53.551518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551427    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e5f9bc-0833-4071-bb36-941e32e98430" containerName="maas-api"
Oct 30 18:03:53.551518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.551436    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d6e400-bca3-48cc-a421-8b356760cc5a" containerName="revoke-keys"
Oct 30 18:03:53.556916 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.556894    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.575855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.575816    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"]
Oct 30 18:03:53.581674 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.581617    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:03:53.581861 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.581679    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:03:53.625321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.625275    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.625532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.625438    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxrg\" (UniqueName: \"kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.625716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.625690    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.677887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.677854    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:53.727509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.727462    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.727805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.727537    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.727805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.727621    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxrg\" (UniqueName: \"kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.728299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.728276    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.730838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.730815    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.751169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.751145    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxrg\" (UniqueName: \"kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg\") pod \"maas-api-revoke-sub-f3aa0e1b-g5skj\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:53.799684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.799605    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:53.876911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:53.876878    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:54.193559 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:54.193534    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"]
Oct 30 18:03:54.195566 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:03:54.195531    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59afd1d4_2643_466f_97ff_f6bd1fe39c05.slice/crio-8f365cfa39b87cac8693b09ccf9f79ec0b95d736fa02baf9df95e6bf64706a92 WatchSource:0}: Error finding container 8f365cfa39b87cac8693b09ccf9f79ec0b95d736fa02baf9df95e6bf64706a92: Status 404 returned error can't find the container with id 8f365cfa39b87cac8693b09ccf9f79ec0b95d736fa02baf9df95e6bf64706a92
Oct 30 18:03:54.526059 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:54.525960    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj" event={"ID":"59afd1d4-2643-466f-97ff-f6bd1fe39c05","Type":"ContainerStarted","Data":"8f365cfa39b87cac8693b09ccf9f79ec0b95d736fa02baf9df95e6bf64706a92"}
Oct 30 18:03:55.185995 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.185964    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:55.352481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.352451    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:55.527317 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.527219    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"]
Oct 30 18:03:55.527561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.527530    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy" containerID="cri-o://9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada" gracePeriod=30
Oct 30 18:03:55.542985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.542945    5534 generic.go:356] "Generic (PLEG): container finished" podID="59afd1d4-2643-466f-97ff-f6bd1fe39c05" containerID="a4b8c9cbd9a203c07018a105e22812c601a76bac880308cf5456b7f33075f013" exitCode=0
Oct 30 18:03:55.543147 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.543104    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj" event={"ID":"59afd1d4-2643-466f-97ff-f6bd1fe39c05","Type":"ContainerDied","Data":"a4b8c9cbd9a203c07018a105e22812c601a76bac880308cf5456b7f33075f013"}
Oct 30 18:03:55.576710 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.576672    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:55.722547 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:55.722503    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:56.048675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.048610    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"]
Oct 30 18:03:56.053485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.053465    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.072246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.072219    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"]
Oct 30 18:03:56.128931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.128901    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.129122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.128948    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkck5\" (UniqueName: \"kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.129189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.129120    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.231056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.231022    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.231560 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.231065    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkck5\" (UniqueName: \"kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.231560 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.231138    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.231862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.231840    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.233631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.233611    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.254876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.254845    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkck5\" (UniqueName: \"kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.391549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.391517    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:56.721340 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:03:56.721310    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f53cd70_e615_4a9f_a714_c48f504e1f86.slice/crio-12f63fb2dbb456ee5efca5cad27b362fac6efcb8177e9e93fee90df0a08a1625 WatchSource:0}: Error finding container 12f63fb2dbb456ee5efca5cad27b362fac6efcb8177e9e93fee90df0a08a1625: Status 404 returned error can't find the container with id 12f63fb2dbb456ee5efca5cad27b362fac6efcb8177e9e93fee90df0a08a1625
Oct 30 18:03:56.721808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.721467    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"]
Oct 30 18:03:56.736228 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.736203    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:56.841126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.841098    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca\") pod \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") "
Oct 30 18:03:56.841307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.841133    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca\") pod \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") "
Oct 30 18:03:56.841352 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.841317    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg\" (UniqueName: \"kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg\") pod \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\" (UID: \"59afd1d4-2643-466f-97ff-f6bd1fe39c05\") "
Oct 30 18:03:56.841551 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.841526    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca" pod "59afd1d4-2643-466f-97ff-f6bd1fe39c05" (UID: "59afd1d4-2643-466f-97ff-f6bd1fe39c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:03:56.841885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.841815    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:56.843556 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.843533    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca" pod "59afd1d4-2643-466f-97ff-f6bd1fe39c05" (UID: "59afd1d4-2643-466f-97ff-f6bd1fe39c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:56.843556 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.843534    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg" pod "59afd1d4-2643-466f-97ff-f6bd1fe39c05" (UID: "59afd1d4-2643-466f-97ff-f6bd1fe39c05"). InnerVolumeSpecName "kube-api-access-qnxrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:56.943583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.943549    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59afd1d4-2643-466f-97ff-f6bd1fe39c05-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:56.943583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:56.943581    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxrg\" (UniqueName: \"kubernetes.io/projected/59afd1d4-2643-466f-97ff-f6bd1fe39c05-kube-api-access-qnxrg\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:57.569569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.569466    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl" event={"ID":"6f53cd70-e615-4a9f-a714-c48f504e1f86","Type":"ContainerStarted","Data":"170b3e2855ee732e8ce29d41bd7e10434187ada24b428e91d2cfc76a9146848a"}
Oct 30 18:03:57.569569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.569547    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl" event={"ID":"6f53cd70-e615-4a9f-a714-c48f504e1f86","Type":"ContainerStarted","Data":"12f63fb2dbb456ee5efca5cad27b362fac6efcb8177e9e93fee90df0a08a1625"}
Oct 30 18:03:57.571520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.571495    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"
Oct 30 18:03:57.571706 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.571537    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj" event={"ID":"59afd1d4-2643-466f-97ff-f6bd1fe39c05","Type":"ContainerDied","Data":"8f365cfa39b87cac8693b09ccf9f79ec0b95d736fa02baf9df95e6bf64706a92"}
Oct 30 18:03:57.571706 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.571580    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f365cfa39b87cac8693b09ccf9f79ec0b95d736fa02baf9df95e6bf64706a92"
Oct 30 18:03:57.603103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.602827    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:57.800432 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:57.800378    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl" podStartSLOduration=1.800360722 podStartE2EDuration="1.800360722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:57.611875346 +0000 UTC m=+3043.373919160" watchObservedRunningTime="2026-10-30 18:03:57.800360722 +0000 UTC m=+3043.562404536"
Oct 30 18:03:58.582575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:58.582540    5534 generic.go:356] "Generic (PLEG): container finished" podID="6f53cd70-e615-4a9f-a714-c48f504e1f86" containerID="170b3e2855ee732e8ce29d41bd7e10434187ada24b428e91d2cfc76a9146848a" exitCode=0
Oct 30 18:03:58.583020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:58.582633    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl" event={"ID":"6f53cd70-e615-4a9f-a714-c48f504e1f86","Type":"ContainerDied","Data":"170b3e2855ee732e8ce29d41bd7e10434187ada24b428e91d2cfc76a9146848a"}
Oct 30 18:03:59.072547 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.072512    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:59.197257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.197178    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:59.471667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.457872    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"]
Oct 30 18:03:59.471667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.459230    5534 cpu_manager.go:414] "containerMap: removing container" podUID="59afd1d4-2643-466f-97ff-f6bd1fe39c05" containerName="revoke-keys"
Oct 30 18:03:59.471667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.459252    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59afd1d4-2643-466f-97ff-f6bd1fe39c05" containerName="revoke-keys"
Oct 30 18:03:59.471667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.459465    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="59afd1d4-2643-466f-97ff-f6bd1fe39c05" containerName="revoke-keys"
Oct 30 18:03:59.493666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.485974    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.506877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.506847    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f10dda\""
Oct 30 18:03:59.507229 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.507212    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f10dda-dockercfg-782rp\""
Oct 30 18:03:59.517293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.515707    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"]
Oct 30 18:03:59.602253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.602213    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.602805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.602392    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsxt2\" (UniqueName: \"kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.703530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.703492    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.703815 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.703606    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsxt2\" (UniqueName: \"kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.704341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.704317    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.727953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.727858    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsxt2\" (UniqueName: \"kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2\") pod \"payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.774267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.774233    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:03:59.804692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.804609    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5\" (UniqueName: \"kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5\") pod \"6f53cd70-e615-4a9f-a714-c48f504e1f86\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") "
Oct 30 18:03:59.804897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.804724    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca\") pod \"6f53cd70-e615-4a9f-a714-c48f504e1f86\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") "
Oct 30 18:03:59.808272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.808236    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca" pod "6f53cd70-e615-4a9f-a714-c48f504e1f86" (UID: "6f53cd70-e615-4a9f-a714-c48f504e1f86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:03:59.808438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.808284    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5" pod "6f53cd70-e615-4a9f-a714-c48f504e1f86" (UID: "6f53cd70-e615-4a9f-a714-c48f504e1f86"). InnerVolumeSpecName "kube-api-access-qkck5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:03:59.858208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.858176    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:03:59.874491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.874436    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:03:59.906591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.906078    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca\") pod \"6f53cd70-e615-4a9f-a714-c48f504e1f86\" (UID: \"6f53cd70-e615-4a9f-a714-c48f504e1f86\") "
Oct 30 18:03:59.906591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.906461    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkck5\" (UniqueName: \"kubernetes.io/projected/6f53cd70-e615-4a9f-a714-c48f504e1f86-kube-api-access-qkck5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:59.906591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.906476    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:03:59.906591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:03:59.906540    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca" pod "6f53cd70-e615-4a9f-a714-c48f504e1f86" (UID: "6f53cd70-e615-4a9f-a714-c48f504e1f86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:00.008636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.008555    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f53cd70-e615-4a9f-a714-c48f504e1f86-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:00.202532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.202499    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"]
Oct 30 18:04:00.203796 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:00.203770    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da177a2_1e9a_4c22_8fba_dc58d448815c.slice/crio-de9ab8b2692803ae047278f002d5c8eba77d3ee2c4f88ce2068dd5e6d57f7d32 WatchSource:0}: Error finding container de9ab8b2692803ae047278f002d5c8eba77d3ee2c4f88ce2068dd5e6d57f7d32: Status 404 returned error can't find the container with id de9ab8b2692803ae047278f002d5c8eba77d3ee2c4f88ce2068dd5e6d57f7d32
Oct 30 18:04:00.473774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.473745    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:00.604986 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.604920    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"
Oct 30 18:04:00.604986 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.604962    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl" event={"ID":"6f53cd70-e615-4a9f-a714-c48f504e1f86","Type":"ContainerDied","Data":"12f63fb2dbb456ee5efca5cad27b362fac6efcb8177e9e93fee90df0a08a1625"}
Oct 30 18:04:00.605535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.605023    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12f63fb2dbb456ee5efca5cad27b362fac6efcb8177e9e93fee90df0a08a1625"
Oct 30 18:04:00.605535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.605140    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:00.607060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.607033    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85" event={"ID":"3da177a2-1e9a-4c22-8fba-dc58d448815c","Type":"ContainerStarted","Data":"6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c"}
Oct 30 18:04:00.607192 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.607083    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85" event={"ID":"3da177a2-1e9a-4c22-8fba-dc58d448815c","Type":"ContainerStarted","Data":"de9ab8b2692803ae047278f002d5c8eba77d3ee2c4f88ce2068dd5e6d57f7d32"}
Oct 30 18:04:00.608002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.607982    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:04:00.637343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.637317    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"]
Oct 30 18:04:00.642976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.642953    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"]
Oct 30 18:04:00.643191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.643168    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy" containerID="cri-o://83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793" gracePeriod=30
Oct 30 18:04:00.673692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.670761    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-90b48a-mgxb6"]
Oct 30 18:04:00.673692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.673336    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85" podStartSLOduration=1.673320758 podStartE2EDuration="1.673320758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:00.668825332 +0000 UTC m=+3046.430869148" watchObservedRunningTime="2026-10-30 18:04:00.673320758 +0000 UTC m=+3046.435364571"
Oct 30 18:04:00.720996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.720950    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:00.932539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:00.932518    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 18:04:01.031338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031301    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031438    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031499    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031583    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031630    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031681    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:01.031738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031708    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031747    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031804    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031921    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v\") pod \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\" (UID: \"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a\") "
Oct 30 18:04:01.031957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031941    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:01.032238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.031991    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:01.032814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032412    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:01.032814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032624    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:01.032814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032783    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.032814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032806    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.033106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032820    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.033106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032835    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.033106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.032848    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.035103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.035064    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:04:01.035256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.035230    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "kube-api-access-qn96v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:01.036020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.036000    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:01.036098 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.036052    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token" pod "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" (UID: "a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:01.128046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.127965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:01.133807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.133782    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.133807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.133807    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.133973 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.133817    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.133973 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.133826    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn96v\" (UniqueName: \"kubernetes.io/projected/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a-kube-api-access-qn96v\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:01.617765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.617734    5534 generic.go:356] "Generic (PLEG): container finished" podID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerID="9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada" exitCode=0
Oct 30 18:04:01.618225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.617803    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"
Oct 30 18:04:01.618225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.617813    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" event={"ID":"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a","Type":"ContainerDied","Data":"9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada"}
Oct 30 18:04:01.618225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.617897    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw" event={"ID":"a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a","Type":"ContainerDied","Data":"83fb750f4f66e315f20d6053d2103020ea6b2f523d06f580659c67994e664374"}
Oct 30 18:04:01.618225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.617921    5534 scope.go:122] "RemoveContainer" containerID="9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada"
Oct 30 18:04:01.632974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.632957    5534 scope.go:122] "RemoveContainer" containerID="c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"
Oct 30 18:04:01.643582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.643563    5534 scope.go:122] "RemoveContainer" containerID="9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada"
Oct 30 18:04:01.643901 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:01.643876    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada\": container with ID starting with 9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada not found: ID does not exist" containerID="9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada"
Oct 30 18:04:01.643976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.643911    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada"} err="failed to get container status \"9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada\": rpc error: code = NotFound desc = could not find container \"9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada\": container with ID starting with 9a84d59055325ebc0f9ba096285e5a41fde4aaf757b3f9751813ac65422fdada not found: ID does not exist"
Oct 30 18:04:01.643976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.643931    5534 scope.go:122] "RemoveContainer" containerID="c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"
Oct 30 18:04:01.644224 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:01.644207    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e\": container with ID starting with c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e not found: ID does not exist" containerID="c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"
Oct 30 18:04:01.644269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.644228    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e"} err="failed to get container status \"c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e\": rpc error: code = NotFound desc = could not find container \"c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e\": container with ID starting with c5ff77f85b7f8404ae1db8c7e1285bc5c1a96d8703a7091292727fc73be1f60e not found: ID does not exist"
Oct 30 18:04:01.675443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.675411    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"]
Oct 30 18:04:01.689078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:01.689048    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e927d7-openshift-default-578b9d68b9-n57xw"]
Oct 30 18:04:02.376699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:02.376612    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367a0e22-06df-4ede-b159-0636e793712d" path="/var/lib/kubelet/pods/367a0e22-06df-4ede-b159-0636e793712d/volumes"
Oct 30 18:04:02.377379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:02.377359    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" path="/var/lib/kubelet/pods/a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a/volumes"
Oct 30 18:04:02.790332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:02.790243    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:03.930534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.930502    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"]
Oct 30 18:04:03.931289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931274    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6f53cd70-e615-4a9f-a714-c48f504e1f86" containerName="revoke-keys"
Oct 30 18:04:03.931335 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931291    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f53cd70-e615-4a9f-a714-c48f504e1f86" containerName="revoke-keys"
Oct 30 18:04:03.931335 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931324    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy"
Oct 30 18:04:03.931335 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931330    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy"
Oct 30 18:04:03.931443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931339    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy"
Oct 30 18:04:03.931443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931344    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy"
Oct 30 18:04:03.931509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931464    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f53cd70-e615-4a9f-a714-c48f504e1f86" containerName="revoke-keys"
Oct 30 18:04:03.931509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931476    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy"
Oct 30 18:04:03.931509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.931483    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dcfb39-d037-4a65-b9ca-ef8c1b5fac7a" containerName="istio-proxy"
Oct 30 18:04:03.937187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.937166    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:03.957342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.957319    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"]
Oct 30 18:04:03.962870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.962848    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:04:03.963036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.962905    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:04:03.984787 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.984754    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xgk\" (UniqueName: \"kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:03.985043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.985023    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:03.985149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:03.985123    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.032102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.032065    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"]
Oct 30 18:04:04.036049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.036028    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a74ffa-5kdgk"]
Oct 30 18:04:04.086494 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.086460    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xgk\" (UniqueName: \"kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.086729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.086630    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.086813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.086756    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.087554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.087530    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.089161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.089137    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.110120 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.110099    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xgk\" (UniqueName: \"kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk\") pod \"maas-api-revoke-sub-efa1a826-95srh\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.254413 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.254314    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:04.400584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.398040    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ee91de-cf0e-4a76-94c0-d222b5549189" path="/var/lib/kubelet/pods/c1ee91de-cf0e-4a76-94c0-d222b5549189/volumes"
Oct 30 18:04:04.564960 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:04.564926    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb544957_3dfb_4e72_9a0e_22fdecba6088.slice/crio-f13b59de60050bd68aee27d82f363b0da7e1175629d399143288bd38f5c70b84 WatchSource:0}: Error finding container f13b59de60050bd68aee27d82f363b0da7e1175629d399143288bd38f5c70b84: Status 404 returned error can't find the container with id f13b59de60050bd68aee27d82f363b0da7e1175629d399143288bd38f5c70b84
Oct 30 18:04:04.566816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.566793    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"]
Oct 30 18:04:04.648882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:04.648840    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh" event={"ID":"bb544957-3dfb-4e72-9a0e-22fdecba6088","Type":"ContainerStarted","Data":"f13b59de60050bd68aee27d82f363b0da7e1175629d399143288bd38f5c70b84"}
Oct 30 18:04:05.199431 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:05.199335    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:05.663660 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:05.663326    5534 generic.go:356] "Generic (PLEG): container finished" podID="bb544957-3dfb-4e72-9a0e-22fdecba6088" containerID="f1ab903ec789b27a03e7e416a010280993771ccba37053be26ca3fc711a5c793" exitCode=0
Oct 30 18:04:05.663660 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:05.663395    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh" event={"ID":"bb544957-3dfb-4e72-9a0e-22fdecba6088","Type":"ContainerDied","Data":"f1ab903ec789b27a03e7e416a010280993771ccba37053be26ca3fc711a5c793"}
Oct 30 18:04:05.963935 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:05.963904    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 18:04:06.020809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.020777    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.021005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.020835    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.021005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.020928    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.021005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.020986    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.021202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021071    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.021202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021109    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:06.021202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021146    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.021202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021195    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021215    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021294    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021344    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t\") pod \"a478a115-837d-42ec-a5d8-128378c3483d\" (UID: \"a478a115-837d-42ec-a5d8-128378c3483d\") "
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021504    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021617    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.021670    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.022171    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.022188    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.022199    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a478a115-837d-42ec-a5d8-128378c3483d-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.022207    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.022449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.022219    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.023933 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.023912    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:06.024037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.023964    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:04:06.024425 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.024402    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "kube-api-access-n954t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:06.030909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.030874    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy" pod "a478a115-837d-42ec-a5d8-128378c3483d" (UID: "a478a115-837d-42ec-a5d8-128378c3483d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:06.127854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.127817    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n954t\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-kube-api-access-n954t\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.128107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.128091    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a478a115-837d-42ec-a5d8-128378c3483d-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.128213 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.128196    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a478a115-837d-42ec-a5d8-128378c3483d-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.128213 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.128214    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a478a115-837d-42ec-a5d8-128378c3483d-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:06.559418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.559382    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"]
Oct 30 18:04:06.663443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.663346    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:06.705666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.684695    5534 generic.go:356] "Generic (PLEG): container finished" podID="a478a115-837d-42ec-a5d8-128378c3483d" containerID="83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793" exitCode=0
Oct 30 18:04:06.705666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.685224    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"
Oct 30 18:04:06.705666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.692080    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" event={"ID":"a478a115-837d-42ec-a5d8-128378c3483d","Type":"ContainerDied","Data":"83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793"}
Oct 30 18:04:06.705666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.692156    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn" event={"ID":"a478a115-837d-42ec-a5d8-128378c3483d","Type":"ContainerDied","Data":"078afd616c12ba925d0a0b2d6bdd095c37cb2e5fae1dc3541a38ac2c8c0f8b2a"}
Oct 30 18:04:06.705666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.692185    5534 scope.go:122] "RemoveContainer" containerID="83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793"
Oct 30 18:04:06.727120 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.727093    5534 scope.go:122] "RemoveContainer" containerID="83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793"
Oct 30 18:04:06.727518 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:06.727494    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793\": container with ID starting with 83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793 not found: ID does not exist" containerID="83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793"
Oct 30 18:04:06.727566 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.727531    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793"} err="failed to get container status \"83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793\": rpc error: code = NotFound desc = could not find container \"83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793\": container with ID starting with 83b6fb9e3be7dfe854de69b3f297f7e615045b61fac65c85c8e8c9dd4c964793 not found: ID does not exist"
Oct 30 18:04:06.748894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.748856    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"]
Oct 30 18:04:06.798103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.798069    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:06.800665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.800473    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-jl2kn"]
Oct 30 18:04:06.935747 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.935389    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:06.990900 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:06.990864    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:07.041628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.041600    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk\" (UniqueName: \"kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk\") pod \"bb544957-3dfb-4e72-9a0e-22fdecba6088\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") "
Oct 30 18:04:07.041847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.041674    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca\") pod \"bb544957-3dfb-4e72-9a0e-22fdecba6088\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") "
Oct 30 18:04:07.041847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.041721    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca\") pod \"bb544957-3dfb-4e72-9a0e-22fdecba6088\" (UID: \"bb544957-3dfb-4e72-9a0e-22fdecba6088\") "
Oct 30 18:04:07.042182 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.042155    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca" pod "bb544957-3dfb-4e72-9a0e-22fdecba6088" (UID: "bb544957-3dfb-4e72-9a0e-22fdecba6088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:07.042535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.042519    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:07.044314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.044288    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk" pod "bb544957-3dfb-4e72-9a0e-22fdecba6088" (UID: "bb544957-3dfb-4e72-9a0e-22fdecba6088"). InnerVolumeSpecName "kube-api-access-s8xgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:07.044457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.044352    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca" pod "bb544957-3dfb-4e72-9a0e-22fdecba6088" (UID: "bb544957-3dfb-4e72-9a0e-22fdecba6088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:07.143554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.143524    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xgk\" (UniqueName: \"kubernetes.io/projected/bb544957-3dfb-4e72-9a0e-22fdecba6088-kube-api-access-s8xgk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:07.143554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.143554    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb544957-3dfb-4e72-9a0e-22fdecba6088-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:07.697013 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.696976    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh" event={"ID":"bb544957-3dfb-4e72-9a0e-22fdecba6088","Type":"ContainerDied","Data":"f13b59de60050bd68aee27d82f363b0da7e1175629d399143288bd38f5c70b84"}
Oct 30 18:04:07.697517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.697037    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13b59de60050bd68aee27d82f363b0da7e1175629d399143288bd38f5c70b84"
Oct 30 18:04:07.697517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.697103    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"
Oct 30 18:04:07.733865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.733827    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:07.849405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:07.849374    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:08.088006 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.087904    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"]
Oct 30 18:04:08.088240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.088190    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy" containerID="cri-o://762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738" gracePeriod=30
Oct 30 18:04:08.114684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.114634    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:08.375694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.374784    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a478a115-837d-42ec-a5d8-128378c3483d" path="/var/lib/kubelet/pods/a478a115-837d-42ec-a5d8-128378c3483d/volumes"
Oct 30 18:04:08.567901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.567869    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"]
Oct 30 18:04:08.569031 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.569006    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bb544957-3dfb-4e72-9a0e-22fdecba6088" containerName="revoke-keys"
Oct 30 18:04:08.569031 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.569030    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb544957-3dfb-4e72-9a0e-22fdecba6088" containerName="revoke-keys"
Oct 30 18:04:08.569247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.569050    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy"
Oct 30 18:04:08.569247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.569057    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy"
Oct 30 18:04:08.569356 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.569271    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb544957-3dfb-4e72-9a0e-22fdecba6088" containerName="revoke-keys"
Oct 30 18:04:08.569356 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.569289    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a478a115-837d-42ec-a5d8-128378c3483d" containerName="istio-proxy"
Oct 30 18:04:08.573916 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.573896    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.597429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.597399    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"]
Oct 30 18:04:08.631541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.631439    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-96c514-serving-cert\""
Oct 30 18:04:08.672237 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.672203    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:08.695686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.695635    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"]
Oct 30 18:04:08.700723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.700702    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.710172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.710147    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.710286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.710178    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5nl\" (UniqueName: \"kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.710325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.710307    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.710366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.710323    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.719806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.719777    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"]
Oct 30 18:04:08.731957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.731923    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:04:08.732328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.732312    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:04:08.818784 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.817511    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:08.819577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819438    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.819577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819483    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nl\" (UniqueName: \"kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.819825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819585    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.819825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819781    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.819825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819820    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.819990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819947    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.820051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.819995    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8mj\" (UniqueName: \"kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.822127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.822068    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.824853 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.824830    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.825236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.825216    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.842906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.842882    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5nl\" (UniqueName: \"kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl\") pod \"maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.889028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.888951    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:08.921720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.921693    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.921882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.921834    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.921882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.921877    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8mj\" (UniqueName: \"kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.923978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.923383    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.928770 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.928706    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:08.948010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:08.947950    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8mj\" (UniqueName: \"kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:09.018738 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.018706    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:09.234179 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:09.234149    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b8209a_7697_4c09_b040_42398290a1d4.slice/crio-a74374e68f29e29b5d72411aeab2011eb80d59d9ac272dc8c4b41b305eab6340 WatchSource:0}: Error finding container a74374e68f29e29b5d72411aeab2011eb80d59d9ac272dc8c4b41b305eab6340: Status 404 returned error can't find the container with id a74374e68f29e29b5d72411aeab2011eb80d59d9ac272dc8c4b41b305eab6340
Oct 30 18:04:09.234570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.234545    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"]
Oct 30 18:04:09.335051 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:09.335022    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b4bfd4_cfb7_4b37_ae15_3c87e4bd76b8.slice/crio-6dbc2519a779e03797c2686c8b32de57b05c5f192fa2c9cdbe1a89d325e7bd67 WatchSource:0}: Error finding container 6dbc2519a779e03797c2686c8b32de57b05c5f192fa2c9cdbe1a89d325e7bd67: Status 404 returned error can't find the container with id 6dbc2519a779e03797c2686c8b32de57b05c5f192fa2c9cdbe1a89d325e7bd67
Oct 30 18:04:09.336137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.336114    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"]
Oct 30 18:04:09.719849 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.719800    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt" event={"ID":"50b8209a-7697-4c09-b040-42398290a1d4","Type":"ContainerStarted","Data":"0d9a60971c79154b746d173a649095eb70165eda26204175f1f6d68c4dd137b6"}
Oct 30 18:04:09.720332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.719870    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt" event={"ID":"50b8209a-7697-4c09-b040-42398290a1d4","Type":"ContainerStarted","Data":"a74374e68f29e29b5d72411aeab2011eb80d59d9ac272dc8c4b41b305eab6340"}
Oct 30 18:04:09.720332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.719899    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:09.721637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.721607    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899" event={"ID":"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8","Type":"ContainerStarted","Data":"6dbc2519a779e03797c2686c8b32de57b05c5f192fa2c9cdbe1a89d325e7bd67"}
Oct 30 18:04:09.765609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:09.765561    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt" podStartSLOduration=1.374638854 podStartE2EDuration="1.765546654s" totalImagesPullingTime="390.9078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:09.763035074 +0000 UTC m=+3055.525078927" watchObservedRunningTime="2026-10-30 18:04:09.765546654 +0000 UTC m=+3055.527590466"
Oct 30 18:04:10.731890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:10.731793    5534 generic.go:356] "Generic (PLEG): container finished" podID="a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" containerID="6afa0145d56097871467cb799815a2c13e6f7821b64e9ebed8202d6ffb199555" exitCode=0
Oct 30 18:04:10.731890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:10.731874    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899" event={"ID":"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8","Type":"ContainerDied","Data":"6afa0145d56097871467cb799815a2c13e6f7821b64e9ebed8202d6ffb199555"}
Oct 30 18:04:10.889669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:10.889599    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:11.560354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:11.560298    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerName="payload-pre-processing" containerID="cri-o://dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b" gracePeriod=25
Oct 30 18:04:11.975797 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:11.975765    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:12.073855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.073768    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca\") pod \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") "
Oct 30 18:04:12.074066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.074050    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca\") pod \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") "
Oct 30 18:04:12.074136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.074097    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj\" (UniqueName: \"kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj\") pod \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\" (UID: \"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8\") "
Oct 30 18:04:12.074468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.074435    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca" pod "a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" (UID: "a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:12.074800 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.074741    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:12.076337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.076309    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj" pod "a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" (UID: "a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8"). InnerVolumeSpecName "kube-api-access-sv8mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:12.076561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.076535    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca" pod "a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" (UID: "a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:12.176371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.176333    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv8mj\" (UniqueName: \"kubernetes.io/projected/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-kube-api-access-sv8mj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:12.176371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.176376    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:12.631683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.631562    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:04:12.753025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.752993    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"
Oct 30 18:04:12.753025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.753017    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899" event={"ID":"a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8","Type":"ContainerDied","Data":"6dbc2519a779e03797c2686c8b32de57b05c5f192fa2c9cdbe1a89d325e7bd67"}
Oct 30 18:04:12.753307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:12.753069    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dbc2519a779e03797c2686c8b32de57b05c5f192fa2c9cdbe1a89d325e7bd67"
Oct 30 18:04:13.311683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.311415    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:13.433245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.433219    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:04:13.496072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496041    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.496263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496087    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.496263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496108    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.496403 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496369    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.496497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496476    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:13.496604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496590    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:13.496678 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496528    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:13.496740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496702    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.496740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496736    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.496851 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.496792    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.497046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.497020    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:13.499742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.497462    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.499892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.499788    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket\") pod \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\" (UID: \"96da3fe9-965f-46c8-a94a-e9bf4ada3839\") "
Oct 30 18:04:13.501256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.500536    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.501256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.500559    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.501256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.500574    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.501256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.500586    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.501256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.500801    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:13.503136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.499479    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "kube-api-access-fvvxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:13.509458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.508037    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:13.509458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.508785    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:04:13.509458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.509422    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token" pod "96da3fe9-965f-46c8-a94a-e9bf4ada3839" (UID: "96da3fe9-965f-46c8-a94a-e9bf4ada3839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:13.602281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.602250    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.602281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.602279    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.602281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.602287    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.602533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.602296    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96da3fe9-965f-46c8-a94a-e9bf4ada3839-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.602533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.602305    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvvxn\" (UniqueName: \"kubernetes.io/projected/96da3fe9-965f-46c8-a94a-e9bf4ada3839-kube-api-access-fvvxn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:13.767269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.767173    5534 generic.go:356] "Generic (PLEG): container finished" podID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerID="762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738" exitCode=0
Oct 30 18:04:13.767269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.767226    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"
Oct 30 18:04:13.767269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.767251    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" event={"ID":"96da3fe9-965f-46c8-a94a-e9bf4ada3839","Type":"ContainerDied","Data":"762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738"}
Oct 30 18:04:13.767530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.767317    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9" event={"ID":"96da3fe9-965f-46c8-a94a-e9bf4ada3839","Type":"ContainerDied","Data":"2cdeef6bee7d37a40d362f674afd73edbdadf6c97cdede85e2a4373d797dc939"}
Oct 30 18:04:13.767530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.767347    5534 scope.go:122] "RemoveContainer" containerID="762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738"
Oct 30 18:04:13.785555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.785532    5534 scope.go:122] "RemoveContainer" containerID="762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738"
Oct 30 18:04:13.785958 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:13.785935    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738\": container with ID starting with 762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738 not found: ID does not exist" containerID="762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738"
Oct 30 18:04:13.786063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.785968    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738"} err="failed to get container status \"762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738\": rpc error: code = NotFound desc = could not find container \"762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738\": container with ID starting with 762b6aaa1c47e2bee136cad64f43417c2a396df9f62b8525f04c6941ab3b3738 not found: ID does not exist"
Oct 30 18:04:13.821248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.821215    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"]
Oct 30 18:04:13.827296 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:13.827271    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-mlpw9"]
Oct 30 18:04:14.373926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:14.373897    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" path="/var/lib/kubelet/pods/96da3fe9-965f-46c8-a94a-e9bf4ada3839/volumes"
Oct 30 18:04:14.926949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:14.926916    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:15.046908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:15.046877    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"]
Oct 30 18:04:15.053182 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:15.053129    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:15.060816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:15.060788    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5538cf0e-bk77q"]
Oct 30 18:04:15.246973 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:15.246887    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:04:15.738837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:15.738807    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:04:16.120361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:16.120324    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:16.376364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:16.376267    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7e10d3-c9a9-48c9-9003-89ca3e003443" path="/var/lib/kubelet/pods/fd7e10d3-c9a9-48c9-9003-89ca3e003443/volumes"
Oct 30 18:04:16.850217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:16.850181    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:17.122276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:17.122172    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:18.013283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.013247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:18.592719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.592684    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:18.917307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.917216    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"]
Oct 30 18:04:18.918410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.918382    5534 cpu_manager.go:414] "containerMap: removing container" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy"
Oct 30 18:04:18.918410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.918404    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy"
Oct 30 18:04:18.918661 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.918448    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" containerName="revoke-keys"
Oct 30 18:04:18.918661 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.918454    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" containerName="revoke-keys"
Oct 30 18:04:18.918661 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.918560    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" containerName="revoke-keys"
Oct 30 18:04:18.918661 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.918572    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="96da3fe9-965f-46c8-a94a-e9bf4ada3839" containerName="istio-proxy"
Oct 30 18:04:18.923975 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.923951    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:18.945655 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.945610    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:04:18.951173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.951148    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:04:18.951878 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:18.951854    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"]
Oct 30 18:04:19.016526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.016483    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.016948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.016559    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5bl\" (UniqueName: \"kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.016948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.016602    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.117913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.117878    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.118122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.117941    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5bl\" (UniqueName: \"kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.118122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.117982    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.118563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.118541    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.120601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.120573    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.141232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.141201    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5bl\" (UniqueName: \"kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl\") pod \"maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.249580 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.249464    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:19.341181 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.341143    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:19.458205 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.458058    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:19.606732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.606701    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"]
Oct 30 18:04:19.606903 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:19.606875    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549cad2a_44b4_4d0a_9a7c_972cd6bb4815.slice/crio-b778305a811396594d46172d1533e5e0de42a6b79f0c6a9faf86367d43ab7752 WatchSource:0}: Error finding container b778305a811396594d46172d1533e5e0de42a6b79f0c6a9faf86367d43ab7752: Status 404 returned error can't find the container with id b778305a811396594d46172d1533e5e0de42a6b79f0c6a9faf86367d43ab7752
Oct 30 18:04:19.872656 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:19.872537    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8" event={"ID":"549cad2a-44b4-4d0a-9a7c-972cd6bb4815","Type":"ContainerStarted","Data":"b778305a811396594d46172d1533e5e0de42a6b79f0c6a9faf86367d43ab7752"}
Oct 30 18:04:20.119107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:20.119074    5534 scope.go:122] "RemoveContainer" containerID="89774dfb7eda014bf944ed504d02a86962d8051cfd78f4a7a842fd24496d9fd8"
Oct 30 18:04:20.142438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:20.142408    5534 scope.go:122] "RemoveContainer" containerID="b6efbfc257877b85d8e08010e4b100aa0734bff26e012269e552d52c1c42d0cf"
Oct 30 18:04:20.154699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:20.154632    5534 scope.go:122] "RemoveContainer" containerID="634bd815790de0cce0e436601221894f58135dc7244a69147b4ee246e15caf84"
Oct 30 18:04:20.506912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:20.506824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:20.885684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:20.885631    5534 generic.go:356] "Generic (PLEG): container finished" podID="549cad2a-44b4-4d0a-9a7c-972cd6bb4815" containerID="74f086ed2866164ac6b842c7c90e939dd7960e90281fc2251c4e553c18c2167d" exitCode=0
Oct 30 18:04:20.885907 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:20.885719    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8" event={"ID":"549cad2a-44b4-4d0a-9a7c-972cd6bb4815","Type":"ContainerDied","Data":"74f086ed2866164ac6b842c7c90e939dd7960e90281fc2251c4e553c18c2167d"}
Oct 30 18:04:22.088503 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.088478    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:22.175202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.175113    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl\" (UniqueName: \"kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl\") pod \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") "
Oct 30 18:04:22.175362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.175312    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca\") pod \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") "
Oct 30 18:04:22.175499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.175483    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca\") pod \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\" (UID: \"549cad2a-44b4-4d0a-9a7c-972cd6bb4815\") "
Oct 30 18:04:22.177024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.176993    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca" pod "549cad2a-44b4-4d0a-9a7c-972cd6bb4815" (UID: "549cad2a-44b4-4d0a-9a7c-972cd6bb4815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:22.179720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.179526    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl" pod "549cad2a-44b4-4d0a-9a7c-972cd6bb4815" (UID: "549cad2a-44b4-4d0a-9a7c-972cd6bb4815"). InnerVolumeSpecName "kube-api-access-qw5bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:22.196020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.195988    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca" pod "549cad2a-44b4-4d0a-9a7c-972cd6bb4815" (UID: "549cad2a-44b4-4d0a-9a7c-972cd6bb4815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:22.204938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.204875    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:22.278024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.277992    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw5bl\" (UniqueName: \"kubernetes.io/projected/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-kube-api-access-qw5bl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:22.278024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.278022    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:22.278250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.278033    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549cad2a-44b4-4d0a-9a7c-972cd6bb4815-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:22.918308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.918279    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"
Oct 30 18:04:22.918868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.918377    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8" event={"ID":"549cad2a-44b4-4d0a-9a7c-972cd6bb4815","Type":"ContainerDied","Data":"b778305a811396594d46172d1533e5e0de42a6b79f0c6a9faf86367d43ab7752"}
Oct 30 18:04:22.919010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:22.918924    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b778305a811396594d46172d1533e5e0de42a6b79f0c6a9faf86367d43ab7752"
Oct 30 18:04:23.405144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:23.405101    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:24.079394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:24.079353    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:25.030998 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:25.030924    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"]
Oct 30 18:04:25.042637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:25.042596    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d916ec-6j8mf"]
Oct 30 18:04:25.512656 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:25.512597    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:26.120554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.120495    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:26.143806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.143771    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:26.271725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.271689    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:26.318313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.318275    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"]
Oct 30 18:04:26.340953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.340910    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-738845-6g899"]
Oct 30 18:04:26.373243 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.373149    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b6b390-7f85-4a10-b29f-aa2b21a28815" path="/var/lib/kubelet/pods/09b6b390-7f85-4a10-b29f-aa2b21a28815/volumes"
Oct 30 18:04:26.373793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.373771    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8" path="/var/lib/kubelet/pods/a0b4bfd4-cfb7-4b37-ae15-3c87e4bd76b8/volumes"
Oct 30 18:04:26.398087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:26.398058    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:27.627153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:27.627117    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:27.759835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:27.759801    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"]
Oct 30 18:04:27.760092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:27.760061    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n" podUID="766c8b4e-6c73-4b5e-a920-178787f6e917" containerName="maas-api" containerID="cri-o://ed01f3ec2e6196111d1e023104a67511332f601508af6d448c200a0bc6601a21" gracePeriod=30
Oct 30 18:04:27.847908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:27.842729    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:27.985339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:27.985299    5534 generic.go:356] "Generic (PLEG): container finished" podID="766c8b4e-6c73-4b5e-a920-178787f6e917" containerID="ed01f3ec2e6196111d1e023104a67511332f601508af6d448c200a0bc6601a21" exitCode=0
Oct 30 18:04:27.985529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:27.985368    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n" event={"ID":"766c8b4e-6c73-4b5e-a920-178787f6e917","Type":"ContainerDied","Data":"ed01f3ec2e6196111d1e023104a67511332f601508af6d448c200a0bc6601a21"}
Oct 30 18:04:28.128768 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.128745    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 18:04:28.179286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.179259    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls\") pod \"766c8b4e-6c73-4b5e-a920-178787f6e917\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") "
Oct 30 18:04:28.179456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.179342    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca\") pod \"766c8b4e-6c73-4b5e-a920-178787f6e917\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") "
Oct 30 18:04:28.179456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.179408    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs\") pod \"766c8b4e-6c73-4b5e-a920-178787f6e917\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") "
Oct 30 18:04:28.179630 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.179607    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz\" (UniqueName: \"kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz\") pod \"766c8b4e-6c73-4b5e-a920-178787f6e917\" (UID: \"766c8b4e-6c73-4b5e-a920-178787f6e917\") "
Oct 30 18:04:28.182339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.182124    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca" pod "766c8b4e-6c73-4b5e-a920-178787f6e917" (UID: "766c8b4e-6c73-4b5e-a920-178787f6e917"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:28.182955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.182926    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz" pod "766c8b4e-6c73-4b5e-a920-178787f6e917" (UID: "766c8b4e-6c73-4b5e-a920-178787f6e917"). InnerVolumeSpecName "kube-api-access-6kxxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:28.185248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.185174    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs" pod "766c8b4e-6c73-4b5e-a920-178787f6e917" (UID: "766c8b4e-6c73-4b5e-a920-178787f6e917"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:28.185362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.185292    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls" pod "766c8b4e-6c73-4b5e-a920-178787f6e917" (UID: "766c8b4e-6c73-4b5e-a920-178787f6e917"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:28.281555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.281510    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/766c8b4e-6c73-4b5e-a920-178787f6e917-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:28.281555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.281544    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:28.281555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.281555    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kxxz\" (UniqueName: \"kubernetes.io/projected/766c8b4e-6c73-4b5e-a920-178787f6e917-kube-api-access-6kxxz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:28.281555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.281564    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/766c8b4e-6c73-4b5e-a920-178787f6e917-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:28.374281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.374247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:28.494725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.494337    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:28.504039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.504012    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"]
Oct 30 18:04:28.504220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.504199    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b" podUID="8773137a-7b6e-4389-aa0f-08221a1f737c" containerName="payload-pre-processing" containerID="cri-o://7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f" gracePeriod=30
Oct 30 18:04:28.624863 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.624829    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:28.645532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.645504    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"]
Oct 30 18:04:28.654065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.654030    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ce0480-w6dxl"]
Oct 30 18:04:28.825062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.825038    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:04:28.924666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.915125    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume\") pod \"8773137a-7b6e-4389-aa0f-08221a1f737c\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") "
Oct 30 18:04:28.924666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.915281    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr\" (UniqueName: \"kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr\") pod \"8773137a-7b6e-4389-aa0f-08221a1f737c\" (UID: \"8773137a-7b6e-4389-aa0f-08221a1f737c\") "
Oct 30 18:04:29.005667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.942848    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr" pod "8773137a-7b6e-4389-aa0f-08221a1f737c" (UID: "8773137a-7b6e-4389-aa0f-08221a1f737c"). InnerVolumeSpecName "kube-api-access-6cnfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:29.005667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:28.956372    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume" pod "8773137a-7b6e-4389-aa0f-08221a1f737c" (UID: "8773137a-7b6e-4389-aa0f-08221a1f737c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:29.049721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.026567    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8773137a-7b6e-4389-aa0f-08221a1f737c-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:29.049721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.026604    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cnfr\" (UniqueName: \"kubernetes.io/projected/8773137a-7b6e-4389-aa0f-08221a1f737c-kube-api-access-6cnfr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:29.132531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.132487    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"]
Oct 30 18:04:29.139927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.139899    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"
Oct 30 18:04:29.146009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.144808    5534 generic.go:356] "Generic (PLEG): container finished" podID="8773137a-7b6e-4389-aa0f-08221a1f737c" containerID="7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f" exitCode=0
Oct 30 18:04:29.146009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.144857    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"
Oct 30 18:04:29.147312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.147291    5534 cpu_manager.go:414] "containerMap: removing container" podUID="766c8b4e-6c73-4b5e-a920-178787f6e917" containerName="maas-api"
Oct 30 18:04:29.147463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.147450    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766c8b4e-6c73-4b5e-a920-178787f6e917" containerName="maas-api"
Oct 30 18:04:29.147583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.147573    5534 cpu_manager.go:414] "containerMap: removing container" podUID="549cad2a-44b4-4d0a-9a7c-972cd6bb4815" containerName="revoke-keys"
Oct 30 18:04:29.147702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.147689    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549cad2a-44b4-4d0a-9a7c-972cd6bb4815" containerName="revoke-keys"
Oct 30 18:04:29.147803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.147793    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8773137a-7b6e-4389-aa0f-08221a1f737c" containerName="payload-pre-processing"
Oct 30 18:04:29.147889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.147877    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8773137a-7b6e-4389-aa0f-08221a1f737c" containerName="payload-pre-processing"
Oct 30 18:04:29.148525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.148508    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="766c8b4e-6c73-4b5e-a920-178787f6e917" containerName="maas-api"
Oct 30 18:04:29.149185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.149170    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="549cad2a-44b4-4d0a-9a7c-972cd6bb4815" containerName="revoke-keys"
Oct 30 18:04:29.149327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.149314    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8773137a-7b6e-4389-aa0f-08221a1f737c" containerName="payload-pre-processing"
Oct 30 18:04:29.159283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.159268    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"]
Oct 30 18:04:29.159450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.159434    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n" event={"ID":"766c8b4e-6c73-4b5e-a920-178787f6e917","Type":"ContainerDied","Data":"2cc274d283df15dcd0a50c8ff4dc00770555d4e4993b8d0baaed14eb45ca03c2"}
Oct 30 18:04:29.159565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.159370    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.159670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.159658    5534 scope.go:122] "RemoveContainer" containerID="ed01f3ec2e6196111d1e023104a67511332f601508af6d448c200a0bc6601a21"
Oct 30 18:04:29.160033 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.159548    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b" event={"ID":"8773137a-7b6e-4389-aa0f-08221a1f737c","Type":"ContainerDied","Data":"7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f"}
Oct 30 18:04:29.160134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.160065    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b" event={"ID":"8773137a-7b6e-4389-aa0f-08221a1f737c","Type":"ContainerDied","Data":"f9e84b8ef59b0476d06f351a05ded622fe2757d2070c8cf5aa382d0df0731258"}
Oct 30 18:04:29.165362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.164405    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:29.176771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.175892    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:04:29.176771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.175994    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:04:29.177923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.177907    5534 scope.go:122] "RemoveContainer" containerID="7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f"
Oct 30 18:04:29.189329 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.189310    5534 scope.go:122] "RemoveContainer" containerID="7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f"
Oct 30 18:04:29.189616 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:29.189599    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f\": container with ID starting with 7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f not found: ID does not exist" containerID="7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f"
Oct 30 18:04:29.189730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.189623    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f"} err="failed to get container status \"7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f\": rpc error: code = NotFound desc = could not find container \"7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f\": container with ID starting with 7e97b0ef6c3f14ee0da97faeb62f8a9bd983058e32f83eaa9f56d3d462cf5b2f not found: ID does not exist"
Oct 30 18:04:29.202116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.202093    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"]
Oct 30 18:04:29.206173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.206152    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ce0480-65978c4dbd-jhr8n"]
Oct 30 18:04:29.239357 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.239333    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.239460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.239370    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.239460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.239394    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstx4\" (UniqueName: \"kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.293951 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.293923    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"]
Oct 30 18:04:29.299685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.299661    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-455e4a-5f7f6d7f-nbh4b"]
Oct 30 18:04:29.340999 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.340971    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.341162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.341007    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.341162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.341032    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstx4\" (UniqueName: \"kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.341666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.341627    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.343501 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.343482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.365395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.365367    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstx4\" (UniqueName: \"kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.375601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.375569    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:29.484373 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.484288    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:29.502975 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.502539    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"]
Oct 30 18:04:29.505748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.505725    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-455e4a-x2rv5"]
Oct 30 18:04:29.539435 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.539399    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:29.835486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:29.835456    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"]
Oct 30 18:04:29.836025 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:29.836001    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916b961e_91f7_4fa5_8b9d_ef564a528eed.slice/crio-6783ba6d8ccb3bdb36d4c3b80d2bb6a3a13c496c751d7d1250c0984030407511 WatchSource:0}: Error finding container 6783ba6d8ccb3bdb36d4c3b80d2bb6a3a13c496c751d7d1250c0984030407511: Status 404 returned error can't find the container with id 6783ba6d8ccb3bdb36d4c3b80d2bb6a3a13c496c751d7d1250c0984030407511
Oct 30 18:04:30.014272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.014238    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:30.157407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.157373    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5" event={"ID":"916b961e-91f7-4fa5-8b9d-ef564a528eed","Type":"ContainerStarted","Data":"6783ba6d8ccb3bdb36d4c3b80d2bb6a3a13c496c751d7d1250c0984030407511"}
Oct 30 18:04:30.378461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.378427    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d6e400-bca3-48cc-a421-8b356760cc5a" path="/var/lib/kubelet/pods/59d6e400-bca3-48cc-a421-8b356760cc5a/volumes"
Oct 30 18:04:30.379155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.379136    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f53cd70-e615-4a9f-a714-c48f504e1f86" path="/var/lib/kubelet/pods/6f53cd70-e615-4a9f-a714-c48f504e1f86/volumes"
Oct 30 18:04:30.379775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.379748    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766c8b4e-6c73-4b5e-a920-178787f6e917" path="/var/lib/kubelet/pods/766c8b4e-6c73-4b5e-a920-178787f6e917/volumes"
Oct 30 18:04:30.380723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.380704    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8773137a-7b6e-4389-aa0f-08221a1f737c" path="/var/lib/kubelet/pods/8773137a-7b6e-4389-aa0f-08221a1f737c/volumes"
Oct 30 18:04:30.447262 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.447189    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"]
Oct 30 18:04:30.477997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.477969    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"]
Oct 30 18:04:30.594292 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.591092    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:30.728406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.728322    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:30.847393 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:30.847354    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:31.202045 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.188451    5534 generic.go:356] "Generic (PLEG): container finished" podID="916b961e-91f7-4fa5-8b9d-ef564a528eed" containerID="b7b6f3e1cfdae40d9082cbc3c5a6e6167d68c747e4415ecda53ca164aa46f09d" exitCode=0
Oct 30 18:04:31.202045 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.188765    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5" event={"ID":"916b961e-91f7-4fa5-8b9d-ef564a528eed","Type":"ContainerDied","Data":"b7b6f3e1cfdae40d9082cbc3c5a6e6167d68c747e4415ecda53ca164aa46f09d"}
Oct 30 18:04:31.462317 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.462290    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:31.903097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.903065    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 18:04:31.991428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.991389    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume\") pod \"f065fa93-9947-4eba-8440-5ead8e5c2188\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") "
Oct 30 18:04:31.991627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.991508    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj\" (UniqueName: \"kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj\") pod \"f065fa93-9947-4eba-8440-5ead8e5c2188\" (UID: \"f065fa93-9947-4eba-8440-5ead8e5c2188\") "
Oct 30 18:04:31.991857 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.991827    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume" pod "f065fa93-9947-4eba-8440-5ead8e5c2188" (UID: "f065fa93-9947-4eba-8440-5ead8e5c2188"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:31.992381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:31.992352    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f065fa93-9947-4eba-8440-5ead8e5c2188-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:32.006871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.006798    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj" pod "f065fa93-9947-4eba-8440-5ead8e5c2188" (UID: "f065fa93-9947-4eba-8440-5ead8e5c2188"). InnerVolumeSpecName "kube-api-access-bqjzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:32.094492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.094450    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjzj\" (UniqueName: \"kubernetes.io/projected/f065fa93-9947-4eba-8440-5ead8e5c2188-kube-api-access-bqjzj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:32.201464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.201426    5534 generic.go:356] "Generic (PLEG): container finished" podID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerID="dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b" exitCode=0
Oct 30 18:04:32.201464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.201471    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"
Oct 30 18:04:32.201753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.201510    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" event={"ID":"f065fa93-9947-4eba-8440-5ead8e5c2188","Type":"ContainerDied","Data":"dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b"}
Oct 30 18:04:32.201753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.201567    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx" event={"ID":"f065fa93-9947-4eba-8440-5ead8e5c2188","Type":"ContainerDied","Data":"adbcc4a63dac5408be6675ea72db95f037ab333d1870378bd79556bdc7e956dc"}
Oct 30 18:04:32.201753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.201594    5534 scope.go:122] "RemoveContainer" containerID="dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b"
Oct 30 18:04:32.215858 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.215834    5534 scope.go:122] "RemoveContainer" containerID="dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b"
Oct 30 18:04:32.216260 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:32.216238    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b\": container with ID starting with dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b not found: ID does not exist" containerID="dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b"
Oct 30 18:04:32.216332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.216269    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b"} err="failed to get container status \"dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b\": rpc error: code = NotFound desc = could not find container \"dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b\": container with ID starting with dfa5c6df4726e0492d199c8d5136e8dea0afee806fe8e519768fb8907282629b not found: ID does not exist"
Oct 30 18:04:32.259913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.259830    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"]
Oct 30 18:04:32.265784 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.265748    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ce0480-58db6c5b67-8hzdx"]
Oct 30 18:04:32.375077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.375049    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" path="/var/lib/kubelet/pods/f065fa93-9947-4eba-8440-5ead8e5c2188/volumes"
Oct 30 18:04:32.402880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.402848    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:32.499565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.499531    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4\" (UniqueName: \"kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4\") pod \"916b961e-91f7-4fa5-8b9d-ef564a528eed\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") "
Oct 30 18:04:32.499762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.499748    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca\") pod \"916b961e-91f7-4fa5-8b9d-ef564a528eed\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") "
Oct 30 18:04:32.499946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.499919    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca\") pod \"916b961e-91f7-4fa5-8b9d-ef564a528eed\" (UID: \"916b961e-91f7-4fa5-8b9d-ef564a528eed\") "
Oct 30 18:04:32.501207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.501178    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca" pod "916b961e-91f7-4fa5-8b9d-ef564a528eed" (UID: "916b961e-91f7-4fa5-8b9d-ef564a528eed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:32.503894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.503867    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca" pod "916b961e-91f7-4fa5-8b9d-ef564a528eed" (UID: "916b961e-91f7-4fa5-8b9d-ef564a528eed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:32.504520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.504447    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4" pod "916b961e-91f7-4fa5-8b9d-ef564a528eed" (UID: "916b961e-91f7-4fa5-8b9d-ef564a528eed"). InnerVolumeSpecName "kube-api-access-vstx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:32.602351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.602318    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:32.602351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.602346    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916b961e-91f7-4fa5-8b9d-ef564a528eed-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:32.602576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:32.602358    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstx4\" (UniqueName: \"kubernetes.io/projected/916b961e-91f7-4fa5-8b9d-ef564a528eed-kube-api-access-vstx4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:33.213180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:33.213152    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"
Oct 30 18:04:33.213654 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:33.213189    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5" event={"ID":"916b961e-91f7-4fa5-8b9d-ef564a528eed","Type":"ContainerDied","Data":"6783ba6d8ccb3bdb36d4c3b80d2bb6a3a13c496c751d7d1250c0984030407511"}
Oct 30 18:04:33.213654 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:33.213238    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6783ba6d8ccb3bdb36d4c3b80d2bb6a3a13c496c751d7d1250c0984030407511"
Oct 30 18:04:33.362804 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:33.362767    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:35.374553 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:35.374523    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:35.447456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:35.447413    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz" podUID="deb7cb4b-06b9-4d0a-9b44-be8addc82864" containerName="payload-processing" containerID="cri-o://47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12" gracePeriod=25
Oct 30 18:04:35.478894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:35.478856    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659" podUID="cba9a1be-9936-47a6-9479-00e5aa827fc8" containerName="payload-pre-processing" containerID="cri-o://db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52" gracePeriod=25
Oct 30 18:04:35.511285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:35.511255    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:35.647753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:35.642264    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:35.783669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:35.779161    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:36.056965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.051688    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"]
Oct 30 18:04:36.109524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.109473    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8dcd3e3-tbz2d"]
Oct 30 18:04:36.117596 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.117570    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:04:36.173893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.173863    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:04:36.222368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.220937    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume\") pod \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") "
Oct 30 18:04:36.222368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.220977    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w\" (UniqueName: \"kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w\") pod \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\" (UID: \"deb7cb4b-06b9-4d0a-9b44-be8addc82864\") "
Oct 30 18:04:36.222368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.222247    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume" pod "deb7cb4b-06b9-4d0a-9b44-be8addc82864" (UID: "deb7cb4b-06b9-4d0a-9b44-be8addc82864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:36.229184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.229147    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w" pod "deb7cb4b-06b9-4d0a-9b44-be8addc82864" (UID: "deb7cb4b-06b9-4d0a-9b44-be8addc82864"). InnerVolumeSpecName "kube-api-access-qfb8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:36.246673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.241996    5534 generic.go:356] "Generic (PLEG): container finished" podID="deb7cb4b-06b9-4d0a-9b44-be8addc82864" containerID="47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12" exitCode=0
Oct 30 18:04:36.246673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.242128    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz" event={"ID":"deb7cb4b-06b9-4d0a-9b44-be8addc82864","Type":"ContainerDied","Data":"47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12"}
Oct 30 18:04:36.246673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.242178    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz" event={"ID":"deb7cb4b-06b9-4d0a-9b44-be8addc82864","Type":"ContainerDied","Data":"e3ccf606f912e90a3b96b2a718ad7ffba904a16580ef89fafa4d086cf99d2d7d"}
Oct 30 18:04:36.246673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.242206    5534 scope.go:122] "RemoveContainer" containerID="47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12"
Oct 30 18:04:36.246673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.242349    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"
Oct 30 18:04:36.250773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.250311    5534 generic.go:356] "Generic (PLEG): container finished" podID="cba9a1be-9936-47a6-9479-00e5aa827fc8" containerID="db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52" exitCode=0
Oct 30 18:04:36.250773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.250416    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"
Oct 30 18:04:36.250773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.250617    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659" event={"ID":"cba9a1be-9936-47a6-9479-00e5aa827fc8","Type":"ContainerDied","Data":"db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52"}
Oct 30 18:04:36.250773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.250730    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659" event={"ID":"cba9a1be-9936-47a6-9479-00e5aa827fc8","Type":"ContainerDied","Data":"323c41f04bbf14ff29e5778356e529d04f9a52a2e047d20da1c19bf33cb728da"}
Oct 30 18:04:36.266681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.266629    5534 scope.go:122] "RemoveContainer" containerID="47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12"
Oct 30 18:04:36.267053 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:36.267027    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12\": container with ID starting with 47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12 not found: ID does not exist" containerID="47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12"
Oct 30 18:04:36.267137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.267068    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12"} err="failed to get container status \"47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12\": rpc error: code = NotFound desc = could not find container \"47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12\": container with ID starting with 47c1dac750bded49880a53eb6cb995a1cf4ea7b3912757e2a0c749e0ea301c12 not found: ID does not exist"
Oct 30 18:04:36.267137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.267091    5534 scope.go:122] "RemoveContainer" containerID="db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52"
Oct 30 18:04:36.269814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.269794    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"]
Oct 30 18:04:36.270511 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270498    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerName="payload-pre-processing"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270513    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerName="payload-pre-processing"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270527    5534 cpu_manager.go:414] "containerMap: removing container" podUID="deb7cb4b-06b9-4d0a-9b44-be8addc82864" containerName="payload-processing"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270532    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb7cb4b-06b9-4d0a-9b44-be8addc82864" containerName="payload-processing"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270545    5534 cpu_manager.go:414] "containerMap: removing container" podUID="916b961e-91f7-4fa5-8b9d-ef564a528eed" containerName="revoke-keys"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270549    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916b961e-91f7-4fa5-8b9d-ef564a528eed" containerName="revoke-keys"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270562    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cba9a1be-9936-47a6-9479-00e5aa827fc8" containerName="payload-pre-processing"
Oct 30 18:04:36.270587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270567    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba9a1be-9936-47a6-9479-00e5aa827fc8" containerName="payload-pre-processing"
Oct 30 18:04:36.271014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270722    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f065fa93-9947-4eba-8440-5ead8e5c2188" containerName="payload-pre-processing"
Oct 30 18:04:36.271014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270731    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb7cb4b-06b9-4d0a-9b44-be8addc82864" containerName="payload-processing"
Oct 30 18:04:36.271014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270739    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="916b961e-91f7-4fa5-8b9d-ef564a528eed" containerName="revoke-keys"
Oct 30 18:04:36.271014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.270745    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba9a1be-9936-47a6-9479-00e5aa827fc8" containerName="payload-pre-processing"
Oct 30 18:04:36.276126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.276106    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.286908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.283011    5534 scope.go:122] "RemoveContainer" containerID="db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52"
Oct 30 18:04:36.286908 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:36.283362    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52\": container with ID starting with db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52 not found: ID does not exist" containerID="db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52"
Oct 30 18:04:36.286908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.283398    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52"} err="failed to get container status \"db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52\": rpc error: code = NotFound desc = could not find container \"db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52\": container with ID starting with db8a57e6158e5fa8c2a2ce351c62fa6d7bbeea69086c2dd7e4d7b3e635950e52 not found: ID does not exist"
Oct 30 18:04:36.311383 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.310724    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:04:36.311383 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.310899    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:04:36.312993 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.312966    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"]
Oct 30 18:04:36.322180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.322150    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume\") pod \"cba9a1be-9936-47a6-9479-00e5aa827fc8\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") "
Oct 30 18:04:36.322416 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.322395    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq\" (UniqueName: \"kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq\") pod \"cba9a1be-9936-47a6-9479-00e5aa827fc8\" (UID: \"cba9a1be-9936-47a6-9479-00e5aa827fc8\") "
Oct 30 18:04:36.323238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.323205    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume" pod "cba9a1be-9936-47a6-9479-00e5aa827fc8" (UID: "cba9a1be-9936-47a6-9479-00e5aa827fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:36.326092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.326066    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq" pod "cba9a1be-9936-47a6-9479-00e5aa827fc8" (UID: "cba9a1be-9936-47a6-9479-00e5aa827fc8"). InnerVolumeSpecName "kube-api-access-4dprq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:36.331714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.331695    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"]
Oct 30 18:04:36.332150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.332130    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba9a1be-9936-47a6-9479-00e5aa827fc8-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:36.332251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.332155    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dprq\" (UniqueName: \"kubernetes.io/projected/cba9a1be-9936-47a6-9479-00e5aa827fc8-kube-api-access-4dprq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:36.332251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.332170    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb7cb4b-06b9-4d0a-9b44-be8addc82864-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:36.332251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.332185    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfb8w\" (UniqueName: \"kubernetes.io/projected/deb7cb4b-06b9-4d0a-9b44-be8addc82864-kube-api-access-qfb8w\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:36.335888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.335868    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-6c89c877c6-jdzqz"]
Oct 30 18:04:36.373502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.373468    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d400f341-6511-4e7e-b5bf-dce70dc44bba" path="/var/lib/kubelet/pods/d400f341-6511-4e7e-b5bf-dce70dc44bba/volumes"
Oct 30 18:04:36.373924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.373908    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb7cb4b-06b9-4d0a-9b44-be8addc82864" path="/var/lib/kubelet/pods/deb7cb4b-06b9-4d0a-9b44-be8addc82864/volumes"
Oct 30 18:04:36.433310 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.433281    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.433780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.433758    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzg5\" (UniqueName: \"kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.433899 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.433883    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.535601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.535564    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.535808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.535751    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.535947 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.535932    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzg5\" (UniqueName: \"kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.536716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.536679    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.538072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.538049    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.559013 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.558986    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzg5\" (UniqueName: \"kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.599448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.598208    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"]
Oct 30 18:04:36.605051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.605021    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:36.605202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.605189    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-57857c647f-vv659"]
Oct 30 18:04:36.924862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:36.924832    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"]
Oct 30 18:04:36.925621 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:36.925593    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d885e3_b2ca_4856_9490_1253b0aa5849.slice/crio-d35077fca78d434dcf5e3975d700f3317d31123100fdba9893808b0828fcb24d WatchSource:0}: Error finding container d35077fca78d434dcf5e3975d700f3317d31123100fdba9893808b0828fcb24d: Status 404 returned error can't find the container with id d35077fca78d434dcf5e3975d700f3317d31123100fdba9893808b0828fcb24d
Oct 30 18:04:37.258252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.258135    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:37.262400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.262374    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm" event={"ID":"27d885e3-b2ca-4856-9490-1253b0aa5849","Type":"ContainerStarted","Data":"d35077fca78d434dcf5e3975d700f3317d31123100fdba9893808b0828fcb24d"}
Oct 30 18:04:37.398333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.398303    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:37.511391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.511311    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:37.633683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.633621    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"]
Oct 30 18:04:37.633944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.633910    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy" containerID="cri-o://d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8" gracePeriod=30
Oct 30 18:04:37.668985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:37.668958    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:38.173907 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.173878    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"]
Oct 30 18:04:38.183591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.179081    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.201685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.201459    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"]
Oct 30 18:04:38.265839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.265806    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgb7v\" (UniqueName: \"kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.265996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.265967    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.266278 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.266254    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.275956 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.275921    5534 generic.go:356] "Generic (PLEG): container finished" podID="27d885e3-b2ca-4856-9490-1253b0aa5849" containerID="d351b1d826dc3e733d165fff36938b6dda2e18e20f5eb278edcccd62642696fd" exitCode=0
Oct 30 18:04:38.276098 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.275971    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm" event={"ID":"27d885e3-b2ca-4856-9490-1253b0aa5849","Type":"ContainerDied","Data":"d351b1d826dc3e733d165fff36938b6dda2e18e20f5eb278edcccd62642696fd"}
Oct 30 18:04:38.367046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.367019    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.367216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.367096    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgb7v\" (UniqueName: \"kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.367216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.367150    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.367852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.367822    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.369763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.369737    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.374321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.374296    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba9a1be-9936-47a6-9479-00e5aa827fc8" path="/var/lib/kubelet/pods/cba9a1be-9936-47a6-9479-00e5aa827fc8/volumes"
Oct 30 18:04:38.393339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.393317    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgb7v\" (UniqueName: \"kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.501239 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.501162    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:38.840158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:38.840125    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"]
Oct 30 18:04:38.840718 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:38.840690    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode557992d_cc4a_4b98_a482_c2714f651369.slice/crio-1687f0fa2bd6fc4528cad1ecb6fed4de4fdaacfcb6a96a6ad927694c8a7d3a11 WatchSource:0}: Error finding container 1687f0fa2bd6fc4528cad1ecb6fed4de4fdaacfcb6a96a6ad927694c8a7d3a11: Status 404 returned error can't find the container with id 1687f0fa2bd6fc4528cad1ecb6fed4de4fdaacfcb6a96a6ad927694c8a7d3a11
Oct 30 18:04:39.288272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.288230    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt" event={"ID":"e557992d-cc4a-4b98-a482-c2714f651369","Type":"ContainerStarted","Data":"1687f0fa2bd6fc4528cad1ecb6fed4de4fdaacfcb6a96a6ad927694c8a7d3a11"}
Oct 30 18:04:39.464805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.464780    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:39.511072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.511039    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"]
Oct 30 18:04:39.520790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.520759    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:39.540919 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.540881    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"]
Oct 30 18:04:39.588262 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.588217    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5\" (UniqueName: \"kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5\") pod \"27d885e3-b2ca-4856-9490-1253b0aa5849\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") "
Oct 30 18:04:39.588475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.588361    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca\") pod \"27d885e3-b2ca-4856-9490-1253b0aa5849\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") "
Oct 30 18:04:39.588701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.588676    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca\") pod \"27d885e3-b2ca-4856-9490-1253b0aa5849\" (UID: \"27d885e3-b2ca-4856-9490-1253b0aa5849\") "
Oct 30 18:04:39.591295 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.590992    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca" pod "27d885e3-b2ca-4856-9490-1253b0aa5849" (UID: "27d885e3-b2ca-4856-9490-1253b0aa5849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:39.592286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.592262    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5" pod "27d885e3-b2ca-4856-9490-1253b0aa5849" (UID: "27d885e3-b2ca-4856-9490-1253b0aa5849"). InnerVolumeSpecName "kube-api-access-jdzg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:39.599070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.593150    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca" pod "27d885e3-b2ca-4856-9490-1253b0aa5849" (UID: "27d885e3-b2ca-4856-9490-1253b0aa5849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:39.650735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.650690    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:39.691143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.691054    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdzg5\" (UniqueName: \"kubernetes.io/projected/27d885e3-b2ca-4856-9490-1253b0aa5849-kube-api-access-jdzg5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:39.691143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.691085    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:39.691143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.691098    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d885e3-b2ca-4856-9490-1253b0aa5849-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:39.776533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:39.776494    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:40.298433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:40.298346    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"
Oct 30 18:04:40.298433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:40.298393    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm" event={"ID":"27d885e3-b2ca-4856-9490-1253b0aa5849","Type":"ContainerDied","Data":"d35077fca78d434dcf5e3975d700f3317d31123100fdba9893808b0828fcb24d"}
Oct 30 18:04:40.298949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:40.298451    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35077fca78d434dcf5e3975d700f3317d31123100fdba9893808b0828fcb24d"
Oct 30 18:04:40.300427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:40.300404    5534 generic.go:356] "Generic (PLEG): container finished" podID="e557992d-cc4a-4b98-a482-c2714f651369" containerID="d20c6ce7beeb471a3440bc04f7047174ec4f084a8c601bb0eb8c5a295a8f5946" exitCode=0
Oct 30 18:04:40.300559 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:40.300491    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt" event={"ID":"e557992d-cc4a-4b98-a482-c2714f651369","Type":"ContainerDied","Data":"d20c6ce7beeb471a3440bc04f7047174ec4f084a8c601bb0eb8c5a295a8f5946"}
Oct 30 18:04:41.499621 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.499598    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:41.627825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.627793    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca\") pod \"e557992d-cc4a-4b98-a482-c2714f651369\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") "
Oct 30 18:04:41.628038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.627980    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca\") pod \"e557992d-cc4a-4b98-a482-c2714f651369\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") "
Oct 30 18:04:41.628133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.628114    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v\" (UniqueName: \"kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v\") pod \"e557992d-cc4a-4b98-a482-c2714f651369\" (UID: \"e557992d-cc4a-4b98-a482-c2714f651369\") "
Oct 30 18:04:41.628397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.628367    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca" pod "e557992d-cc4a-4b98-a482-c2714f651369" (UID: "e557992d-cc4a-4b98-a482-c2714f651369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:41.628960 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.628934    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:41.630169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.630146    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca" pod "e557992d-cc4a-4b98-a482-c2714f651369" (UID: "e557992d-cc4a-4b98-a482-c2714f651369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:41.630799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.630781    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v" pod "e557992d-cc4a-4b98-a482-c2714f651369" (UID: "e557992d-cc4a-4b98-a482-c2714f651369"). InnerVolumeSpecName "kube-api-access-dgb7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:41.664382 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.664354    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:41.730247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.730213    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgb7v\" (UniqueName: \"kubernetes.io/projected/e557992d-cc4a-4b98-a482-c2714f651369-kube-api-access-dgb7v\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:41.730247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:41.730243    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e557992d-cc4a-4b98-a482-c2714f651369-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:42.320449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.320418    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"
Oct 30 18:04:42.320627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.320464    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt" event={"ID":"e557992d-cc4a-4b98-a482-c2714f651369","Type":"ContainerDied","Data":"1687f0fa2bd6fc4528cad1ecb6fed4de4fdaacfcb6a96a6ad927694c8a7d3a11"}
Oct 30 18:04:42.320627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.320526    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1687f0fa2bd6fc4528cad1ecb6fed4de4fdaacfcb6a96a6ad927694c8a7d3a11"
Oct 30 18:04:42.947591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.947568    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 18:04:42.958757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.958726    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.958927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.958780    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.958927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.958861    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.958927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.958914    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.959102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.958957    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.959102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959000    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.959102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959028    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.959102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959064    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:42.959102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959096    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.959378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959152    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:42.959378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959315    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo\") pod \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\" (UID: \"cbe01f86-f520-4fb1-b7c3-9348cf833e9e\") "
Oct 30 18:04:42.959486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959410    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:42.959825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.959701    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:42.960081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.960059    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:42.960160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.960085    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:42.960160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.960100    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:42.960160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.960113    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:42.960688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.960637    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:42.961731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.961706    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:42.961992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.961966    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:42.962550 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.962522    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "kube-api-access-sgzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:42.963088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:42.963062    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo" pod "cbe01f86-f520-4fb1-b7c3-9348cf833e9e" (UID: "cbe01f86-f520-4fb1-b7c3-9348cf833e9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:04:43.061598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.061564    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzj5\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-kube-api-access-sgzj5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:43.061598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.061594    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:43.061598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.061604    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:43.061859 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.061613    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:43.061859 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.061622    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe01f86-f520-4fb1-b7c3-9348cf833e9e-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:43.331372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.331284    5534 generic.go:356] "Generic (PLEG): container finished" podID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerID="d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8" exitCode=0
Oct 30 18:04:43.331372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.331328    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"
Oct 30 18:04:43.331555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.331365    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" event={"ID":"cbe01f86-f520-4fb1-b7c3-9348cf833e9e","Type":"ContainerDied","Data":"d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8"}
Oct 30 18:04:43.331555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.331427    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk" event={"ID":"cbe01f86-f520-4fb1-b7c3-9348cf833e9e","Type":"ContainerDied","Data":"960bd1a5052f6f7350d97ae26687fff65b10997b0a3e2fd844c599153fbed91d"}
Oct 30 18:04:43.331555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.331456    5534 scope.go:122] "RemoveContainer" containerID="d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8"
Oct 30 18:04:43.347206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.347185    5534 scope.go:122] "RemoveContainer" containerID="20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"
Oct 30 18:04:43.360879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.360768    5534 scope.go:122] "RemoveContainer" containerID="d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8"
Oct 30 18:04:43.361134 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:43.361105    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8\": container with ID starting with d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8 not found: ID does not exist" containerID="d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8"
Oct 30 18:04:43.361217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.361145    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8"} err="failed to get container status \"d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8\": rpc error: code = NotFound desc = could not find container \"d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8\": container with ID starting with d8bdb4318cb1d2cd509542d285b94c7df626937b535535c33e54bb1cc455aba8 not found: ID does not exist"
Oct 30 18:04:43.361217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.361167    5534 scope.go:122] "RemoveContainer" containerID="20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"
Oct 30 18:04:43.361493 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:43.361460    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8\": container with ID starting with 20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8 not found: ID does not exist" containerID="20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"
Oct 30 18:04:43.361580 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.361499    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8"} err="failed to get container status \"20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8\": rpc error: code = NotFound desc = could not find container \"20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8\": container with ID starting with 20401a86cc7986fc68463fafa4e4a362e99fa7f7a4a57df30babbb82051781d8 not found: ID does not exist"
Oct 30 18:04:43.403217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.403176    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"]
Oct 30 18:04:43.407991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:43.407958    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-455e4a-openshift-default-665c968d79-5csmk"]
Oct 30 18:04:44.021108 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:44.021068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:44.372551 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:44.372516    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" path="/var/lib/kubelet/pods/cbe01f86-f520-4fb1-b7c3-9348cf833e9e/volumes"
Oct 30 18:04:44.519668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:44.515779    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerName="payload-processing" containerID="cri-o://131fbc09f84bb6c7926e9533a1b44db550ae9239b0dc28a47d040f97d9d5209b" gracePeriod=25
Oct 30 18:04:44.553668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:44.547714    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" containerName="payload-pre-processing" containerID="cri-o://21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10" gracePeriod=25
Oct 30 18:04:45.246515 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:45.246478    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:04:45.614284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:45.614244    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:45.614497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:45.614244    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:45.989020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:45.988945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:46.709675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:46.709622    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"]
Oct 30 18:04:46.834841 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:46.834803    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:46.956209 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:46.956174    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:47.713423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:47.713388    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:48.584300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:48.584252    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" podUID="cd29d125f430f9e0a8a7b7b6580a8dec" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:04:49.033264 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:49.033229    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"]
Oct 30 18:04:49.044629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:49.044594    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8099e35f-fnlcv"]
Oct 30 18:04:49.477795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:49.477762    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:49.766362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:49.766250    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:49.879391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:49.879359    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:50.373101 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:50.373068    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6f8544-9941-48cd-942d-7d758c85da06" path="/var/lib/kubelet/pods/5a6f8544-9941-48cd-942d-7d758c85da06/volumes"
Oct 30 18:04:51.710236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:51.710185    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerName="payload-processing" containerID="cri-o://c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326" gracePeriod=25
Oct 30 18:04:51.816319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:51.816286    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.028915 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.028884    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.141847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.141813    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.202684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.201474    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"]
Oct 30 18:04:53.202684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.201871    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" podUID="94d38a2b-3e96-471a-aed1-c55be0699e11" containerName="maas-api" containerID="cri-o://0ba94c0d295eb1e53042bf332c5c83a103b3f1a8691eb4f8db17b90418a47fba" gracePeriod=30
Oct 30 18:04:53.263737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.263699    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.339833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.339795    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" podUID="94d38a2b-3e96-471a-aed1-c55be0699e11" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.64:8443/health\": dial tcp 10.133.1.64:8443: connect: connection refused"
Oct 30 18:04:53.358588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.358537    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"]
Oct 30 18:04:53.359659 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.359107    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy" containerID="cri-o://48bb2676d5a0c63f714434762114b5958b79db1511400eae3db7245d699bd3d3" gracePeriod=30
Oct 30 18:04:53.381713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.381662    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"]
Oct 30 18:04:53.382096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.381890    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt" podUID="f70c6ad7-716e-4fd0-8c28-eef78f8543e8" containerName="payload-processing" containerID="cri-o://e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785" gracePeriod=30
Oct 30 18:04:53.418200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.418154    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.484937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.483630    5534 generic.go:356] "Generic (PLEG): container finished" podID="94d38a2b-3e96-471a-aed1-c55be0699e11" containerID="0ba94c0d295eb1e53042bf332c5c83a103b3f1a8691eb4f8db17b90418a47fba" exitCode=0
Oct 30 18:04:53.484937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.483680    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" event={"ID":"94d38a2b-3e96-471a-aed1-c55be0699e11","Type":"ContainerDied","Data":"0ba94c0d295eb1e53042bf332c5c83a103b3f1a8691eb4f8db17b90418a47fba"}
Oct 30 18:04:53.550159 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.550110    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.620886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.620607    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:04:53.750596 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.750565    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:04:53.784928 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.784894    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl\" (UniqueName: \"kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl\") pod \"94d38a2b-3e96-471a-aed1-c55be0699e11\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") "
Oct 30 18:04:53.785151 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.785058    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca\") pod \"94d38a2b-3e96-471a-aed1-c55be0699e11\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") "
Oct 30 18:04:53.785222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.785205    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs\") pod \"94d38a2b-3e96-471a-aed1-c55be0699e11\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") "
Oct 30 18:04:53.785333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.785316    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls\") pod \"94d38a2b-3e96-471a-aed1-c55be0699e11\" (UID: \"94d38a2b-3e96-471a-aed1-c55be0699e11\") "
Oct 30 18:04:53.785452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.785423    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca" pod "94d38a2b-3e96-471a-aed1-c55be0699e11" (UID: "94d38a2b-3e96-471a-aed1-c55be0699e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:53.786220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.786191    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d38a2b-3e96-471a-aed1-c55be0699e11-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:53.787552 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.787525    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs" pod "94d38a2b-3e96-471a-aed1-c55be0699e11" (UID: "94d38a2b-3e96-471a-aed1-c55be0699e11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:53.787728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.787611    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls" pod "94d38a2b-3e96-471a-aed1-c55be0699e11" (UID: "94d38a2b-3e96-471a-aed1-c55be0699e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:53.787949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.787926    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl" pod "94d38a2b-3e96-471a-aed1-c55be0699e11" (UID: "94d38a2b-3e96-471a-aed1-c55be0699e11"). InnerVolumeSpecName "kube-api-access-t6rfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:53.929796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.929696    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:53.931048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.931018    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume\") pod \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") "
Oct 30 18:04:53.931224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.931203    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h\" (UniqueName: \"kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h\") pod \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\" (UID: \"f70c6ad7-716e-4fd0-8c28-eef78f8543e8\") "
Oct 30 18:04:53.932068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.931969    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:53.932068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.931998    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d38a2b-3e96-471a-aed1-c55be0699e11-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:53.932068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.932012    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6rfl\" (UniqueName: \"kubernetes.io/projected/94d38a2b-3e96-471a-aed1-c55be0699e11-kube-api-access-t6rfl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:53.934552 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.934284    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume" pod "f70c6ad7-716e-4fd0-8c28-eef78f8543e8" (UID: "f70c6ad7-716e-4fd0-8c28-eef78f8543e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:53.952709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:53.952610    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h" pod "f70c6ad7-716e-4fd0-8c28-eef78f8543e8" (UID: "f70c6ad7-716e-4fd0-8c28-eef78f8543e8"). InnerVolumeSpecName "kube-api-access-sl92h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:54.034039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.034003    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:54.034039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.034036    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl92h\" (UniqueName: \"kubernetes.io/projected/f70c6ad7-716e-4fd0-8c28-eef78f8543e8-kube-api-access-sl92h\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:54.055850 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.055818    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:54.064394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.064353    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"]
Oct 30 18:04:54.068122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.068094    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5cbb4-9pnzt"]
Oct 30 18:04:54.153306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.153269    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"]
Oct 30 18:04:54.154469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154438    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy"
Oct 30 18:04:54.154469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154467    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154483    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f70c6ad7-716e-4fd0-8c28-eef78f8543e8" containerName="payload-processing"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154491    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70c6ad7-716e-4fd0-8c28-eef78f8543e8" containerName="payload-processing"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154526    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e557992d-cc4a-4b98-a482-c2714f651369" containerName="revoke-keys"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154533    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e557992d-cc4a-4b98-a482-c2714f651369" containerName="revoke-keys"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154550    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154557    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154565    5534 cpu_manager.go:414] "containerMap: removing container" podUID="27d885e3-b2ca-4856-9490-1253b0aa5849" containerName="revoke-keys"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154572    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d885e3-b2ca-4856-9490-1253b0aa5849" containerName="revoke-keys"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154592    5534 cpu_manager.go:414] "containerMap: removing container" podUID="94d38a2b-3e96-471a-aed1-c55be0699e11" containerName="maas-api"
Oct 30 18:04:54.154732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154599    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d38a2b-3e96-471a-aed1-c55be0699e11" containerName="maas-api"
Oct 30 18:04:54.155339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154846    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy"
Oct 30 18:04:54.155339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154859    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70c6ad7-716e-4fd0-8c28-eef78f8543e8" containerName="payload-processing"
Oct 30 18:04:54.155339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154873    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e557992d-cc4a-4b98-a482-c2714f651369" containerName="revoke-keys"
Oct 30 18:04:54.155339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154881    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d885e3-b2ca-4856-9490-1253b0aa5849" containerName="revoke-keys"
Oct 30 18:04:54.155339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.154895    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d38a2b-3e96-471a-aed1-c55be0699e11" containerName="maas-api"
Oct 30 18:04:54.159574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.159553    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.168780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.168720    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:54.169463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.169439    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"]
Oct 30 18:04:54.188026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.187955    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:04:54.188206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.187975    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:04:54.339392 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.339360    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.339581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.339550    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdr9\" (UniqueName: \"kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.339709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.339692    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.374685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.374624    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e557992d-cc4a-4b98-a482-c2714f651369" path="/var/lib/kubelet/pods/e557992d-cc4a-4b98-a482-c2714f651369/volumes"
Oct 30 18:04:54.441068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.440991    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdr9\" (UniqueName: \"kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.441247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.441121    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.441247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.441200    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.441862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.441834    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.443994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.443975    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.464536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.464501    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdr9\" (UniqueName: \"kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9\") pod \"maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.479331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.479299    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:54.496482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.495831    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"
Oct 30 18:04:54.496482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.495840    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx" event={"ID":"94d38a2b-3e96-471a-aed1-c55be0699e11","Type":"ContainerDied","Data":"cda51d0ffbc59a7fa1b6570c2586f63044547d89431af8c00274b305d1ab4ce8"}
Oct 30 18:04:54.496482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.495979    5534 scope.go:122] "RemoveContainer" containerID="0ba94c0d295eb1e53042bf332c5c83a103b3f1a8691eb4f8db17b90418a47fba"
Oct 30 18:04:54.497934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.497905    5534 generic.go:356] "Generic (PLEG): container finished" podID="f70c6ad7-716e-4fd0-8c28-eef78f8543e8" containerID="e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785" exitCode=0
Oct 30 18:04:54.498142 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.498125    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt" event={"ID":"f70c6ad7-716e-4fd0-8c28-eef78f8543e8","Type":"ContainerDied","Data":"e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785"}
Oct 30 18:04:54.498222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.498170    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt" event={"ID":"f70c6ad7-716e-4fd0-8c28-eef78f8543e8","Type":"ContainerDied","Data":"c9bacbde3e4d14345737988200b1718354499c602b603f51e2a8f1158b49ddae"}
Oct 30 18:04:54.498487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.498237    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"
Oct 30 18:04:54.528932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.528913    5534 scope.go:122] "RemoveContainer" containerID="e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785"
Oct 30 18:04:54.543848 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.541472    5534 scope.go:122] "RemoveContainer" containerID="e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785"
Oct 30 18:04:54.547160 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:04:54.547128    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785\": container with ID starting with e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785 not found: ID does not exist" containerID="e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785"
Oct 30 18:04:54.547416 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.547392    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785"} err="failed to get container status \"e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785\": rpc error: code = NotFound desc = could not find container \"e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785\": container with ID starting with e7cdf532c02837c64cb8cc7cdc8b99a37a6cba29ac3697aee1692cd85e119785 not found: ID does not exist"
Oct 30 18:04:54.585320 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.585248    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"]
Oct 30 18:04:54.591298 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.591271    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5cbb4-6cd9df5d8d-q8csx"]
Oct 30 18:04:54.596893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.596860    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:54.617320 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.617096    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"]
Oct 30 18:04:54.621359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.621332    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b5cbb4-6dfcd59c6d-dqfvt"]
Oct 30 18:04:54.797619 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:54.797572    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"]
Oct 30 18:04:54.797941 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:04:54.797920    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dceebab_1851_4341_9bb5_1951096b720f.slice/crio-70485ba9ffc67560b40b5537bc4dc5d88492148634aa3f045cec8fd394b7368c WatchSource:0}: Error finding container 70485ba9ffc67560b40b5537bc4dc5d88492148634aa3f045cec8fd394b7368c: Status 404 returned error can't find the container with id 70485ba9ffc67560b40b5537bc4dc5d88492148634aa3f045cec8fd394b7368c
Oct 30 18:04:55.511386 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:55.511284    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x" event={"ID":"5dceebab-1851-4341-9bb5-1951096b720f","Type":"ContainerStarted","Data":"e3243778c9eb4796b376e0dc1eb71907c61518474e00211aa6cb97e844869d95"}
Oct 30 18:04:55.511386 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:55.511349    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x" event={"ID":"5dceebab-1851-4341-9bb5-1951096b720f","Type":"ContainerStarted","Data":"70485ba9ffc67560b40b5537bc4dc5d88492148634aa3f045cec8fd394b7368c"}
Oct 30 18:04:55.613981 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:55.613944    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:55.614374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:55.614349    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:56.132785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:56.132737    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:56.400730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:56.400691    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d38a2b-3e96-471a-aed1-c55be0699e11" path="/var/lib/kubelet/pods/94d38a2b-3e96-471a-aed1-c55be0699e11/volumes"
Oct 30 18:04:56.401145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:56.401129    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70c6ad7-716e-4fd0-8c28-eef78f8543e8" path="/var/lib/kubelet/pods/f70c6ad7-716e-4fd0-8c28-eef78f8543e8/volumes"
Oct 30 18:04:56.524232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:56.524194    5534 generic.go:356] "Generic (PLEG): container finished" podID="5dceebab-1851-4341-9bb5-1951096b720f" containerID="e3243778c9eb4796b376e0dc1eb71907c61518474e00211aa6cb97e844869d95" exitCode=0
Oct 30 18:04:56.524694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:56.524281    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x" event={"ID":"5dceebab-1851-4341-9bb5-1951096b720f","Type":"ContainerDied","Data":"e3243778c9eb4796b376e0dc1eb71907c61518474e00211aa6cb97e844869d95"}
Oct 30 18:04:57.727727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.727703    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:57.813503 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.813409    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca\") pod \"5dceebab-1851-4341-9bb5-1951096b720f\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") "
Oct 30 18:04:57.813716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.813678    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9\" (UniqueName: \"kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9\") pod \"5dceebab-1851-4341-9bb5-1951096b720f\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") "
Oct 30 18:04:57.813832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.813793    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca\") pod \"5dceebab-1851-4341-9bb5-1951096b720f\" (UID: \"5dceebab-1851-4341-9bb5-1951096b720f\") "
Oct 30 18:04:57.814118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.814097    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca" pod "5dceebab-1851-4341-9bb5-1951096b720f" (UID: "5dceebab-1851-4341-9bb5-1951096b720f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:57.814311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.814295    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:57.816225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.816181    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9" pod "5dceebab-1851-4341-9bb5-1951096b720f" (UID: "5dceebab-1851-4341-9bb5-1951096b720f"). InnerVolumeSpecName "kube-api-access-8zdr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:57.816357 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.816267    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca" pod "5dceebab-1851-4341-9bb5-1951096b720f" (UID: "5dceebab-1851-4341-9bb5-1951096b720f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:04:57.915439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.915405    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dceebab-1851-4341-9bb5-1951096b720f-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:57.915439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.915433    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdr9\" (UniqueName: \"kubernetes.io/projected/5dceebab-1851-4341-9bb5-1951096b720f-kube-api-access-8zdr9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:57.920165 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:57.920138    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:58.543473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.543289    5534 generic.go:356] "Generic (PLEG): container finished" podID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerID="48bb2676d5a0c63f714434762114b5958b79db1511400eae3db7245d699bd3d3" exitCode=0
Oct 30 18:04:58.543473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.543362    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" event={"ID":"64d1619a-9925-41f8-a0ec-2e2af4d1bac4","Type":"ContainerDied","Data":"48bb2676d5a0c63f714434762114b5958b79db1511400eae3db7245d699bd3d3"}
Oct 30 18:04:58.545152 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.545132    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"
Oct 30 18:04:58.545273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.545129    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x" event={"ID":"5dceebab-1851-4341-9bb5-1951096b720f","Type":"ContainerDied","Data":"70485ba9ffc67560b40b5537bc4dc5d88492148634aa3f045cec8fd394b7368c"}
Oct 30 18:04:58.545273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.545191    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70485ba9ffc67560b40b5537bc4dc5d88492148634aa3f045cec8fd394b7368c"
Oct 30 18:04:58.642635 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.642331    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:04:58.649991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.649968    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:04:58.724994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.724954    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.725207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725184    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.725307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725253    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.725307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725300    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.725437 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725383    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.725494 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725449    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.726061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725738    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:58.726061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.725760    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:04:58.726061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.726022    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:58.726307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.726160    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.726418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.726396    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:58.726486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.726468    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.726902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.726879    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket\") pod \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\" (UID: \"64d1619a-9925-41f8-a0ec-2e2af4d1bac4\") "
Oct 30 18:04:58.727195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.727144    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:58.728191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.728167    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.728707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.728613    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.728929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.728712    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.728929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.728739    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.728929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.728754    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.729308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.729271    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:58.729423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.729296    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "kube-api-access-9vgbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:04:58.729423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.729393    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:04:58.730851 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.730824    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo" pod "64d1619a-9925-41f8-a0ec-2e2af4d1bac4" (UID: "64d1619a-9925-41f8-a0ec-2e2af4d1bac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:04:58.830574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.830537    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgbq\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-kube-api-access-9vgbq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.830574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.830571    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.830574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.830582    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.830860 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.830591    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d1619a-9925-41f8-a0ec-2e2af4d1bac4-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:04:58.947748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:58.947664    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"]
Oct 30 18:04:59.058392 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.058356    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:59.404283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.404253    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:59.522752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.522718    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"]
Oct 30 18:04:59.522976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.522952    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" podUID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" containerName="payload-pre-processing" containerID="cri-o://c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938" gracePeriod=30
Oct 30 18:04:59.584662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.584616    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"
Oct 30 18:04:59.584662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.584654    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr" event={"ID":"64d1619a-9925-41f8-a0ec-2e2af4d1bac4","Type":"ContainerDied","Data":"d4b5ad52b71d86c1c0abe38e5ceede48f21e5d5c6596729410c0736b8dcde946"}
Oct 30 18:04:59.584913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.584711    5534 scope.go:122] "RemoveContainer" containerID="48bb2676d5a0c63f714434762114b5958b79db1511400eae3db7245d699bd3d3"
Oct 30 18:04:59.646914 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.646879    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:59.647707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.647686    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"]
Oct 30 18:04:59.653472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.653409    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-42mpr"]
Oct 30 18:04:59.840691 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.835009    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:04:59.996754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:04:59.995375    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:05:00.032397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.032362    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"]
Oct 30 18:05:00.038491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.038298    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-196e5b96-v7qm8"]
Oct 30 18:05:00.066331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.066287    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s\" (UniqueName: \"kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s\") pod \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") "
Oct 30 18:05:00.067118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.067078    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume\") pod \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\" (UID: \"69598f5e-08bd-4fac-b74b-c9c30d5867e4\") "
Oct 30 18:05:00.067828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.067684    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume" pod "69598f5e-08bd-4fac-b74b-c9c30d5867e4" (UID: "69598f5e-08bd-4fac-b74b-c9c30d5867e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:00.068553 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.068524    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69598f5e-08bd-4fac-b74b-c9c30d5867e4-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:00.071238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.071200    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s" pod "69598f5e-08bd-4fac-b74b-c9c30d5867e4" (UID: "69598f5e-08bd-4fac-b74b-c9c30d5867e4"). InnerVolumeSpecName "kube-api-access-p768s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:00.170334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.170292    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p768s\" (UniqueName: \"kubernetes.io/projected/69598f5e-08bd-4fac-b74b-c9c30d5867e4-kube-api-access-p768s\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:00.192927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.192897    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:00.334321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.334206    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:00.422011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.421965    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" path="/var/lib/kubelet/pods/64d1619a-9925-41f8-a0ec-2e2af4d1bac4/volumes"
Oct 30 18:05:00.422741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.422615    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c70b37-f87a-4b3b-ac44-d9d075a65ffd" path="/var/lib/kubelet/pods/72c70b37-f87a-4b3b-ac44-d9d075a65ffd/volumes"
Oct 30 18:05:00.423378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.423354    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"]
Oct 30 18:05:00.423489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.423395    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fab25f-gn8t8"]
Oct 30 18:05:00.459842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.459798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:00.595209 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.595172    5534 generic.go:356] "Generic (PLEG): container finished" podID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" containerID="c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938" exitCode=0
Oct 30 18:05:00.595405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.595236    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"
Oct 30 18:05:00.595405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.595256    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" event={"ID":"69598f5e-08bd-4fac-b74b-c9c30d5867e4","Type":"ContainerDied","Data":"c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938"}
Oct 30 18:05:00.595405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.595313    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd" event={"ID":"69598f5e-08bd-4fac-b74b-c9c30d5867e4","Type":"ContainerDied","Data":"21b33d095836031b022228e151e42b5fffaa8e2b820970f50649b92cd1f19f3f"}
Oct 30 18:05:00.595405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.595340    5534 scope.go:122] "RemoveContainer" containerID="c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938"
Oct 30 18:05:00.610863 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.610840    5534 scope.go:122] "RemoveContainer" containerID="c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938"
Oct 30 18:05:00.611194 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:00.611169    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938\": container with ID starting with c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938 not found: ID does not exist" containerID="c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938"
Oct 30 18:05:00.611280 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.611209    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938"} err="failed to get container status \"c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938\": rpc error: code = NotFound desc = could not find container \"c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938\": container with ID starting with c4b2fffd4daffaf5e9105460b96306640511fcf3ec6683b1c5555be1c78a5938 not found: ID does not exist"
Oct 30 18:05:00.670703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.670665    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"]
Oct 30 18:05:00.679912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.679875    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-6b6b67b9c9-7psjd"]
Oct 30 18:05:00.923138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:00.923058    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:01.645504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:01.645471    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:01.804667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:01.794545    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:02.029322 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.029231    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:02.241313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.241277    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:02.372658 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.372611    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549cad2a-44b4-4d0a-9a7c-972cd6bb4815" path="/var/lib/kubelet/pods/549cad2a-44b4-4d0a-9a7c-972cd6bb4815/volumes"
Oct 30 18:05:02.373089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.373069    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" path="/var/lib/kubelet/pods/69598f5e-08bd-4fac-b74b-c9c30d5867e4/volumes"
Oct 30 18:05:02.589306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.589273    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"]
Oct 30 18:05:02.590012 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.589998    5534 cpu_manager.go:414] "containerMap: removing container" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy"
Oct 30 18:05:02.590066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590014    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy"
Oct 30 18:05:02.590066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590032    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5dceebab-1851-4341-9bb5-1951096b720f" containerName="revoke-keys"
Oct 30 18:05:02.590066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590037    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dceebab-1851-4341-9bb5-1951096b720f" containerName="revoke-keys"
Oct 30 18:05:02.590175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590070    5534 cpu_manager.go:414] "containerMap: removing container" podUID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" containerName="payload-pre-processing"
Oct 30 18:05:02.590175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590076    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" containerName="payload-pre-processing"
Oct 30 18:05:02.590175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590175    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe01f86-f520-4fb1-b7c3-9348cf833e9e" containerName="istio-proxy"
Oct 30 18:05:02.590269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590185    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d1619a-9925-41f8-a0ec-2e2af4d1bac4" containerName="istio-proxy"
Oct 30 18:05:02.590269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590193    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dceebab-1851-4341-9bb5-1951096b720f" containerName="revoke-keys"
Oct 30 18:05:02.590269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.590202    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="69598f5e-08bd-4fac-b74b-c9c30d5867e4" containerName="payload-pre-processing"
Oct 30 18:05:02.595188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.595165    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.615191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.615089    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:02.620160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.620136    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"]
Oct 30 18:05:02.627315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.627239    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:02.730597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.730556    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.730809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.730747    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.730888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.730851    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqdt\" (UniqueName: \"kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.831806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.831774    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.831983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.831846    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.831983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.831907    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqdt\" (UniqueName: \"kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.832571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.832544    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.834603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.834578    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.861550 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.861520    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqdt\" (UniqueName: \"kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:02.911250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:02.911139    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:03.227745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.227713    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"]
Oct 30 18:05:03.229504 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:03.229478    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc091550f_1b3e_4c55_b006_21e6adaa8fc8.slice/crio-5945797f0bd9a495c31693444b0253a21874cd8061f73cfa30826554c4728689 WatchSource:0}: Error finding container 5945797f0bd9a495c31693444b0253a21874cd8061f73cfa30826554c4728689: Status 404 returned error can't find the container with id 5945797f0bd9a495c31693444b0253a21874cd8061f73cfa30826554c4728689
Oct 30 18:05:03.528534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.528441    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"]
Oct 30 18:05:03.563776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.563743    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"]
Oct 30 18:05:03.632541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.632498    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx" event={"ID":"c091550f-1b3e-4c55-b006-21e6adaa8fc8","Type":"ContainerStarted","Data":"5945797f0bd9a495c31693444b0253a21874cd8061f73cfa30826554c4728689"}
Oct 30 18:05:03.662577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.662547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:03.777636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.777600    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:03.887303 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:03.887272    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:04.324344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.324311    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"]
Oct 30 18:05:04.371773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.371736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:04.505339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.505305    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:04.637258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.637223    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:04.644232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.644196    5534 generic.go:356] "Generic (PLEG): container finished" podID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerID="131fbc09f84bb6c7926e9533a1b44db550ae9239b0dc28a47d040f97d9d5209b" exitCode=0
Oct 30 18:05:04.644396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.644242    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" event={"ID":"38c17f33-9fb3-42c9-953d-5cd384d1d3a7","Type":"ContainerDied","Data":"131fbc09f84bb6c7926e9533a1b44db550ae9239b0dc28a47d040f97d9d5209b"}
Oct 30 18:05:04.646514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.646483    5534 generic.go:356] "Generic (PLEG): container finished" podID="c091550f-1b3e-4c55-b006-21e6adaa8fc8" containerID="075a2f510cd27d7195d2bf88683b493731ba25c241a17b27aacdb76a45aa0a9b" exitCode=0
Oct 30 18:05:04.646717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.646614    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx" event={"ID":"c091550f-1b3e-4c55-b006-21e6adaa8fc8","Type":"ContainerDied","Data":"075a2f510cd27d7195d2bf88683b493731ba25c241a17b27aacdb76a45aa0a9b"}
Oct 30 18:05:04.766442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.766407    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 18:05:04.878714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.877824    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume\") pod \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") "
Oct 30 18:05:04.878714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.877890    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k\" (UniqueName: \"kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k\") pod \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\" (UID: \"38c17f33-9fb3-42c9-953d-5cd384d1d3a7\") "
Oct 30 18:05:04.879368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.879337    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume" pod "38c17f33-9fb3-42c9-953d-5cd384d1d3a7" (UID: "38c17f33-9fb3-42c9-953d-5cd384d1d3a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:04.882195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.882166    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k" pod "38c17f33-9fb3-42c9-953d-5cd384d1d3a7" (UID: "38c17f33-9fb3-42c9-953d-5cd384d1d3a7"). InnerVolumeSpecName "kube-api-access-cw98k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:04.884551 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.884523    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 18:05:05.013665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.991169    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume\") pod \"df5ecc83-b741-41e6-b818-9133b1feb346\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") "
Oct 30 18:05:05.013665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:04.991285    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd\" (UniqueName: \"kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd\") pod \"df5ecc83-b741-41e6-b818-9133b1feb346\" (UID: \"df5ecc83-b741-41e6-b818-9133b1feb346\") "
Oct 30 18:05:05.049078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.048907    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume" pod "df5ecc83-b741-41e6-b818-9133b1feb346" (UID: "df5ecc83-b741-41e6-b818-9133b1feb346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:05.085670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.080808    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd" pod "df5ecc83-b741-41e6-b818-9133b1feb346" (UID: "df5ecc83-b741-41e6-b818-9133b1feb346"). InnerVolumeSpecName "kube-api-access-2hrdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:05.113450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.112580    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:05.113450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.112622    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw98k\" (UniqueName: \"kubernetes.io/projected/38c17f33-9fb3-42c9-953d-5cd384d1d3a7-kube-api-access-cw98k\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:05.113450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.112637    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ecc83-b741-41e6-b818-9133b1feb346-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:05.113450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.112699    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrdd\" (UniqueName: \"kubernetes.io/projected/df5ecc83-b741-41e6-b818-9133b1feb346-kube-api-access-2hrdd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:05.661703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.661672    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"
Oct 30 18:05:05.661703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.661686    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx" event={"ID":"38c17f33-9fb3-42c9-953d-5cd384d1d3a7","Type":"ContainerDied","Data":"f4f368f337ae5656c613857ccbd3926f307bead3eae544ecb22b6dd6c1f96bc6"}
Oct 30 18:05:05.662302 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.661755    5534 scope.go:122] "RemoveContainer" containerID="131fbc09f84bb6c7926e9533a1b44db550ae9239b0dc28a47d040f97d9d5209b"
Oct 30 18:05:05.663988 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.663952    5534 generic.go:356] "Generic (PLEG): container finished" podID="df5ecc83-b741-41e6-b818-9133b1feb346" containerID="21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10" exitCode=0
Oct 30 18:05:05.664188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.663994    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" event={"ID":"df5ecc83-b741-41e6-b818-9133b1feb346","Type":"ContainerDied","Data":"21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10"}
Oct 30 18:05:05.664188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.664062    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7" event={"ID":"df5ecc83-b741-41e6-b818-9133b1feb346","Type":"ContainerDied","Data":"2ff8946a30783140c212fcb99222283e0bf99b6993a58f38ba94fd975edd55cf"}
Oct 30 18:05:05.664188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.664007    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"
Oct 30 18:05:05.684107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.678687    5534 scope.go:122] "RemoveContainer" containerID="21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10"
Oct 30 18:05:05.692715 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.692693    5534 scope.go:122] "RemoveContainer" containerID="21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10"
Oct 30 18:05:05.693112 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:05.693075    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10\": container with ID starting with 21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10 not found: ID does not exist" containerID="21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10"
Oct 30 18:05:05.693262 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.693115    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10"} err="failed to get container status \"21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10\": rpc error: code = NotFound desc = could not find container \"21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10\": container with ID starting with 21d4a3580e79e87168943665005f7a267b2a25bc380530344ef80f3d6860ab10 not found: ID does not exist"
Oct 30 18:05:05.714532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.714496    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"]
Oct 30 18:05:05.727765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.727728    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-23c1e8-99ccd989d-4mhmx"]
Oct 30 18:05:05.773723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.771755    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"]
Oct 30 18:05:05.775971 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.775933    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-23c1e8-66976dd775-6n2l7"]
Oct 30 18:05:05.873140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.873102    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:05.924084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.924002    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca\") pod \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") "
Oct 30 18:05:05.924272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.924087    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca\") pod \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") "
Oct 30 18:05:05.924272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.924257    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt\" (UniqueName: \"kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt\") pod \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\" (UID: \"c091550f-1b3e-4c55-b006-21e6adaa8fc8\") "
Oct 30 18:05:05.924418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.924391    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca" pod "c091550f-1b3e-4c55-b006-21e6adaa8fc8" (UID: "c091550f-1b3e-4c55-b006-21e6adaa8fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:05.924880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.924843    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:05.927303 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.927267    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca" pod "c091550f-1b3e-4c55-b006-21e6adaa8fc8" (UID: "c091550f-1b3e-4c55-b006-21e6adaa8fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:05.927498 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:05.927476    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt" pod "c091550f-1b3e-4c55-b006-21e6adaa8fc8" (UID: "c091550f-1b3e-4c55-b006-21e6adaa8fc8"). InnerVolumeSpecName "kube-api-access-lzqdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:06.026066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.026024    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqdt\" (UniqueName: \"kubernetes.io/projected/c091550f-1b3e-4c55-b006-21e6adaa8fc8-kube-api-access-lzqdt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:06.026066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.026062    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c091550f-1b3e-4c55-b006-21e6adaa8fc8-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:06.375463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.375423    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" path="/var/lib/kubelet/pods/38c17f33-9fb3-42c9-953d-5cd384d1d3a7/volumes"
Oct 30 18:05:06.376149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.376127    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" path="/var/lib/kubelet/pods/df5ecc83-b741-41e6-b818-9133b1feb346/volumes"
Oct 30 18:05:06.421698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.421628    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:06.679163 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.679064    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"
Oct 30 18:05:06.679163 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.679083    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx" event={"ID":"c091550f-1b3e-4c55-b006-21e6adaa8fc8","Type":"ContainerDied","Data":"5945797f0bd9a495c31693444b0253a21874cd8061f73cfa30826554c4728689"}
Oct 30 18:05:06.679163 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.679131    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5945797f0bd9a495c31693444b0253a21874cd8061f73cfa30826554c4728689"
Oct 30 18:05:06.943618 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:06.943456    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:07.093606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.093574    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:07.391148 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.391107    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"]
Oct 30 18:05:07.391354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.391333    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy" containerID="cri-o://505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da" gracePeriod=30
Oct 30 18:05:07.436934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.436891    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:07.966833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.966788    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"]
Oct 30 18:05:07.967795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967773    5534 cpu_manager.go:414] "containerMap: removing container" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerName="payload-processing"
Oct 30 18:05:07.967795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967796    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerName="payload-processing"
Oct 30 18:05:07.967954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967836    5534 cpu_manager.go:414] "containerMap: removing container" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" containerName="payload-pre-processing"
Oct 30 18:05:07.967954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967842    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" containerName="payload-pre-processing"
Oct 30 18:05:07.967954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967866    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c091550f-1b3e-4c55-b006-21e6adaa8fc8" containerName="revoke-keys"
Oct 30 18:05:07.967954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967872    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c091550f-1b3e-4c55-b006-21e6adaa8fc8" containerName="revoke-keys"
Oct 30 18:05:07.968102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967983    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5ecc83-b741-41e6-b818-9133b1feb346" containerName="payload-pre-processing"
Oct 30 18:05:07.968102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.967994    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c091550f-1b3e-4c55-b006-21e6adaa8fc8" containerName="revoke-keys"
Oct 30 18:05:07.968102 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.968008    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c17f33-9fb3-42c9-953d-5cd384d1d3a7" containerName="payload-processing"
Oct 30 18:05:07.973289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.973261    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:07.993474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:07.993439    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"]
Oct 30 18:05:08.061111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.061071    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrjf\" (UniqueName: \"kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.061350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.061119    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.061350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.061321    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.072451 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.072415    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:08.072679 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.072474    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:08.163452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.163411    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrjf\" (UniqueName: \"kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.163726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.163463    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.163726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.163565    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.164380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.164354    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.166662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.166622    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.187909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.187878    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrjf\" (UniqueName: \"kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.291723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.291616    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:08.344601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.344552    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:08.529355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.529286    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc" podUID="9001bbcd-9da7-4b56-99aa-a255c145ea41" containerName="payload-processing" containerID="cri-o://f81c1db03d3ad148bc0c4cfcc810976dcdfbf4eee2fee7803b08666c289a1b61" gracePeriod=25
Oct 30 18:05:08.567344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.564750    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g" podUID="a491f8f7-bf5f-4240-b398-4d659c16ed39" containerName="payload-pre-processing" containerID="cri-o://a1b2cd035c3f0ffa19bf936511d59865ae9198fc8ac36796ffd2638acde2dadc" gracePeriod=25
Oct 30 18:05:08.614420 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.614381    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"]
Oct 30 18:05:08.642835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.642794    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:05:08.644718 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:08.644690    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce06dbe_7f28_4c9a_ae38_dc712c919980.slice/crio-e41218cf2b321b60aabe1d1b23d9a6a58878221a1eeca22129fe54d20f9197e5 WatchSource:0}: Error finding container e41218cf2b321b60aabe1d1b23d9a6a58878221a1eeca22129fe54d20f9197e5: Status 404 returned error can't find the container with id e41218cf2b321b60aabe1d1b23d9a6a58878221a1eeca22129fe54d20f9197e5
Oct 30 18:05:08.702949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.702636    5534 generic.go:356] "Generic (PLEG): container finished" podID="9001bbcd-9da7-4b56-99aa-a255c145ea41" containerID="f81c1db03d3ad148bc0c4cfcc810976dcdfbf4eee2fee7803b08666c289a1b61" exitCode=0
Oct 30 18:05:08.702949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.702823    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc" event={"ID":"9001bbcd-9da7-4b56-99aa-a255c145ea41","Type":"ContainerDied","Data":"f81c1db03d3ad148bc0c4cfcc810976dcdfbf4eee2fee7803b08666c289a1b61"}
Oct 30 18:05:08.706517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.706305    5534 generic.go:356] "Generic (PLEG): container finished" podID="a491f8f7-bf5f-4240-b398-4d659c16ed39" containerID="a1b2cd035c3f0ffa19bf936511d59865ae9198fc8ac36796ffd2638acde2dadc" exitCode=0
Oct 30 18:05:08.706517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.706450    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g" event={"ID":"a491f8f7-bf5f-4240-b398-4d659c16ed39","Type":"ContainerDied","Data":"a1b2cd035c3f0ffa19bf936511d59865ae9198fc8ac36796ffd2638acde2dadc"}
Oct 30 18:05:08.708289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.708220    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66" event={"ID":"1ce06dbe-7f28-4c9a-ae38-dc712c919980","Type":"ContainerStarted","Data":"e41218cf2b321b60aabe1d1b23d9a6a58878221a1eeca22129fe54d20f9197e5"}
Oct 30 18:05:08.883204 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.883164    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:05:08.942925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.942890    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:05:08.991411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.991368    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume\") pod \"9001bbcd-9da7-4b56-99aa-a255c145ea41\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") "
Oct 30 18:05:08.992000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.991759    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr\" (UniqueName: \"kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr\") pod \"9001bbcd-9da7-4b56-99aa-a255c145ea41\" (UID: \"9001bbcd-9da7-4b56-99aa-a255c145ea41\") "
Oct 30 18:05:08.993564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.993528    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume" pod "9001bbcd-9da7-4b56-99aa-a255c145ea41" (UID: "9001bbcd-9da7-4b56-99aa-a255c145ea41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:08.996952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:08.996914    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr" pod "9001bbcd-9da7-4b56-99aa-a255c145ea41" (UID: "9001bbcd-9da7-4b56-99aa-a255c145ea41"). InnerVolumeSpecName "kube-api-access-tdwtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:09.093880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.093784    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume\") pod \"a491f8f7-bf5f-4240-b398-4d659c16ed39\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") "
Oct 30 18:05:09.093880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.093853    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8\" (UniqueName: \"kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8\") pod \"a491f8f7-bf5f-4240-b398-4d659c16ed39\" (UID: \"a491f8f7-bf5f-4240-b398-4d659c16ed39\") "
Oct 30 18:05:09.094268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.094241    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume" pod "a491f8f7-bf5f-4240-b398-4d659c16ed39" (UID: "a491f8f7-bf5f-4240-b398-4d659c16ed39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:09.094339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.094325    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwtr\" (UniqueName: \"kubernetes.io/projected/9001bbcd-9da7-4b56-99aa-a255c145ea41-kube-api-access-tdwtr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:09.094377 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.094342    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001bbcd-9da7-4b56-99aa-a255c145ea41-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:09.096581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.096508    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8" pod "a491f8f7-bf5f-4240-b398-4d659c16ed39" (UID: "a491f8f7-bf5f-4240-b398-4d659c16ed39"). InnerVolumeSpecName "kube-api-access-pp4w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:09.196067 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.196031    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491f8f7-bf5f-4240-b398-4d659c16ed39-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:09.196067 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.196065    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4w8\" (UniqueName: \"kubernetes.io/projected/a491f8f7-bf5f-4240-b398-4d659c16ed39-kube-api-access-pp4w8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:09.325341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.325280    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerName="payload-processing" containerID="cri-o://38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b" gracePeriod=25
Oct 30 18:05:09.717789 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.717758    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"
Oct 30 18:05:09.718009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.717788    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc" event={"ID":"9001bbcd-9da7-4b56-99aa-a255c145ea41","Type":"ContainerDied","Data":"1a639270ce81c534611fd998c778cf3af78c80a4d09aaefcd84f5816e435eacf"}
Oct 30 18:05:09.718009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.717850    5534 scope.go:122] "RemoveContainer" containerID="f81c1db03d3ad148bc0c4cfcc810976dcdfbf4eee2fee7803b08666c289a1b61"
Oct 30 18:05:09.721472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.721433    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"
Oct 30 18:05:09.721631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.721464    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g" event={"ID":"a491f8f7-bf5f-4240-b398-4d659c16ed39","Type":"ContainerDied","Data":"39eb335588bfb6c6e911229a4b937355ddcf37395d7b768894ecdc7f7e886312"}
Oct 30 18:05:09.723603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.723576    5534 generic.go:356] "Generic (PLEG): container finished" podID="1ce06dbe-7f28-4c9a-ae38-dc712c919980" containerID="8fb7c048e14f211e38fea92328f36a50025a8d3d7c4a71a255b3b6e0b0dc1edb" exitCode=0
Oct 30 18:05:09.723748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.723666    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66" event={"ID":"1ce06dbe-7f28-4c9a-ae38-dc712c919980","Type":"ContainerDied","Data":"8fb7c048e14f211e38fea92328f36a50025a8d3d7c4a71a255b3b6e0b0dc1edb"}
Oct 30 18:05:09.748815 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.748789    5534 scope.go:122] "RemoveContainer" containerID="a1b2cd035c3f0ffa19bf936511d59865ae9198fc8ac36796ffd2638acde2dadc"
Oct 30 18:05:09.839086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.839037    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"]
Oct 30 18:05:09.843660 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.843618    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-68445fdb58-xpfhc"]
Oct 30 18:05:09.879143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.879107    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"]
Oct 30 18:05:09.882832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:09.882802    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-fcb879667-9kr7g"]
Oct 30 18:05:10.248055 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:10.248017    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:10.373151 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:10.373094    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9001bbcd-9da7-4b56-99aa-a255c145ea41" path="/var/lib/kubelet/pods/9001bbcd-9da7-4b56-99aa-a255c145ea41/volumes"
Oct 30 18:05:10.373979 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:10.373955    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a491f8f7-bf5f-4240-b398-4d659c16ed39" path="/var/lib/kubelet/pods/a491f8f7-bf5f-4240-b398-4d659c16ed39/volumes"
Oct 30 18:05:10.921502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:10.921464    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:11.022773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.022676    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca\") pod \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") "
Oct 30 18:05:11.022923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.022778    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca\") pod \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") "
Oct 30 18:05:11.022981 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.022964    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf\" (UniqueName: \"kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf\") pod \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\" (UID: \"1ce06dbe-7f28-4c9a-ae38-dc712c919980\") "
Oct 30 18:05:11.023099 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.023078    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca" pod "1ce06dbe-7f28-4c9a-ae38-dc712c919980" (UID: "1ce06dbe-7f28-4c9a-ae38-dc712c919980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:11.023615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.023588    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:11.025080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.025057    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca" pod "1ce06dbe-7f28-4c9a-ae38-dc712c919980" (UID: "1ce06dbe-7f28-4c9a-ae38-dc712c919980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:11.025318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.025296    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf" pod "1ce06dbe-7f28-4c9a-ae38-dc712c919980" (UID: "1ce06dbe-7f28-4c9a-ae38-dc712c919980"). InnerVolumeSpecName "kube-api-access-vzrjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:11.125354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.125317    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce06dbe-7f28-4c9a-ae38-dc712c919980-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:11.125354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.125351    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrjf\" (UniqueName: \"kubernetes.io/projected/1ce06dbe-7f28-4c9a-ae38-dc712c919980-kube-api-access-vzrjf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:11.750946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.750912    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"
Oct 30 18:05:11.751343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.750910    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66" event={"ID":"1ce06dbe-7f28-4c9a-ae38-dc712c919980","Type":"ContainerDied","Data":"e41218cf2b321b60aabe1d1b23d9a6a58878221a1eeca22129fe54d20f9197e5"}
Oct 30 18:05:11.751343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:11.751065    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41218cf2b321b60aabe1d1b23d9a6a58878221a1eeca22129fe54d20f9197e5"
Oct 30 18:05:12.000118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.000084    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:12.021081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.021056    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 18:05:12.142289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.142253    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb\" (UniqueName: \"kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb\") pod \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") "
Oct 30 18:05:12.142528 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.142508    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume\") pod \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\" (UID: \"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb\") "
Oct 30 18:05:12.143029 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.142997    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume" pod "4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" (UID: "4db1ff6e-79b1-4ef4-83bb-716435f6f1eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:12.143245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.143226    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.144793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.144752    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb" pod "4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" (UID: "4db1ff6e-79b1-4ef4-83bb-716435f6f1eb"). InnerVolumeSpecName "kube-api-access-mrfgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:12.244609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.244518    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrfgb\" (UniqueName: \"kubernetes.io/projected/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb-kube-api-access-mrfgb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.693435 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.693394    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 18:05:12.772900 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.772802    5534 generic.go:356] "Generic (PLEG): container finished" podID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerID="c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326" exitCode=0
Oct 30 18:05:12.774342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.772864    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"
Oct 30 18:05:12.774342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.772892    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" event={"ID":"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb","Type":"ContainerDied","Data":"c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326"}
Oct 30 18:05:12.774342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.773634    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f" event={"ID":"4db1ff6e-79b1-4ef4-83bb-716435f6f1eb","Type":"ContainerDied","Data":"7da2c04ac901defb21f5f5f1319d3904f150ad8c114e19b4dff5c513b9b91b3e"}
Oct 30 18:05:12.774342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.773686    5534 scope.go:122] "RemoveContainer" containerID="c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326"
Oct 30 18:05:12.778727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.777626    5534 generic.go:356] "Generic (PLEG): container finished" podID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerID="505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da" exitCode=0
Oct 30 18:05:12.778727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.777692    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"
Oct 30 18:05:12.778727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.777777    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" event={"ID":"f22b1f6e-24fa-437f-809d-446fbf9f894d","Type":"ContainerDied","Data":"505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da"}
Oct 30 18:05:12.778727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.777821    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj" event={"ID":"f22b1f6e-24fa-437f-809d-446fbf9f894d","Type":"ContainerDied","Data":"aed327488548104cc365fb3fc71efd6827f0706378c5ef7264e37b69c0d83471"}
Oct 30 18:05:12.791325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.791301    5534 scope.go:122] "RemoveContainer" containerID="c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326"
Oct 30 18:05:12.791791 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:12.791763    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326\": container with ID starting with c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326 not found: ID does not exist" containerID="c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326"
Oct 30 18:05:12.791908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.791798    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326"} err="failed to get container status \"c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326\": rpc error: code = NotFound desc = could not find container \"c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326\": container with ID starting with c5089c95bbb289ed00579c0e53cd59131e005b5b4c1a5ccb991c3b1c3dc88326 not found: ID does not exist"
Oct 30 18:05:12.791908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.791823    5534 scope.go:122] "RemoveContainer" containerID="505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da"
Oct 30 18:05:12.803315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.803294    5534 scope.go:122] "RemoveContainer" containerID="505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da"
Oct 30 18:05:12.803785 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:12.803758    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da\": container with ID starting with 505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da not found: ID does not exist" containerID="505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da"
Oct 30 18:05:12.803942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.803917    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da"} err="failed to get container status \"505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da\": rpc error: code = NotFound desc = could not find container \"505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da\": container with ID starting with 505ac8343bf4e5a08a3bd150201a53ba942357302137c57b83e4fd45d3ca50da not found: ID does not exist"
Oct 30 18:05:12.836397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.836355    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"]
Oct 30 18:05:12.841555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.841527    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-01b925-86bc7f66bb-72r8f"]
Oct 30 18:05:12.852438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852409    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852497    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852521    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852577    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852603    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852637    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852745    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852764    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:12.852888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852796    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.852888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852870    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92\") pod \"f22b1f6e-24fa-437f-809d-446fbf9f894d\" (UID: \"f22b1f6e-24fa-437f-809d-446fbf9f894d\") "
Oct 30 18:05:12.853067 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.852921    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:12.853380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853283    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:12.853380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853298    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:12.853605 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853583    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.853717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853611    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.853717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853624    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.853717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853666    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.853872 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.853721    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:12.855283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.855259    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:12.855588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.855562    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:05:12.855711 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.855597    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:12.856071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.856054    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92" pod "f22b1f6e-24fa-437f-809d-446fbf9f894d" (UID: "f22b1f6e-24fa-437f-809d-446fbf9f894d"). InnerVolumeSpecName "kube-api-access-gwv92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:12.955016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.954987    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.955016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.955014    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.955215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.955027    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22b1f6e-24fa-437f-809d-446fbf9f894d-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.955215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.955040    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22b1f6e-24fa-437f-809d-446fbf9f894d-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:12.955215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:12.955050    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwv92\" (UniqueName: \"kubernetes.io/projected/f22b1f6e-24fa-437f-809d-446fbf9f894d-kube-api-access-gwv92\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:13.086962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.086835    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"]
Oct 30 18:05:13.188665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.183603    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"]
Oct 30 18:05:13.194313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.193737    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-xgvgj"]
Oct 30 18:05:13.247893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.247856    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:13.399695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.399629    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:13.541499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.541463    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:13.601106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.601070    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:05:13.706489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.706377    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"]
Oct 30 18:05:13.707224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707201    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a491f8f7-bf5f-4240-b398-4d659c16ed39" containerName="payload-pre-processing"
Oct 30 18:05:13.707285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707228    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a491f8f7-bf5f-4240-b398-4d659c16ed39" containerName="payload-pre-processing"
Oct 30 18:05:13.707285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707258    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy"
Oct 30 18:05:13.707285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707266    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy"
Oct 30 18:05:13.707399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707287    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerName="payload-processing"
Oct 30 18:05:13.707399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707294    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerName="payload-processing"
Oct 30 18:05:13.707399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707331    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9001bbcd-9da7-4b56-99aa-a255c145ea41" containerName="payload-processing"
Oct 30 18:05:13.707399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707337    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9001bbcd-9da7-4b56-99aa-a255c145ea41" containerName="payload-processing"
Oct 30 18:05:13.707399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707368    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1ce06dbe-7f28-4c9a-ae38-dc712c919980" containerName="revoke-keys"
Oct 30 18:05:13.707399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707373    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce06dbe-7f28-4c9a-ae38-dc712c919980" containerName="revoke-keys"
Oct 30 18:05:13.707591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707536    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" containerName="payload-processing"
Oct 30 18:05:13.707591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707549    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9001bbcd-9da7-4b56-99aa-a255c145ea41" containerName="payload-processing"
Oct 30 18:05:13.707591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707560    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce06dbe-7f28-4c9a-ae38-dc712c919980" containerName="revoke-keys"
Oct 30 18:05:13.707591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707571    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a491f8f7-bf5f-4240-b398-4d659c16ed39" containerName="payload-pre-processing"
Oct 30 18:05:13.707591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.707578    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" containerName="istio-proxy"
Oct 30 18:05:13.711191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.711171    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.736171 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.735949    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95f55d\""
Oct 30 18:05:13.736500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.736482    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95f55d-dockercfg-5x7v7\""
Oct 30 18:05:13.740660 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.737753    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"]
Oct 30 18:05:13.797615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.797192    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.797615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.797370    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt54t\" (UniqueName: \"kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.917672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.914903    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt54t\" (UniqueName: \"kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.917672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.915122    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.917672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.916017    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.928664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.925002    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"]
Oct 30 18:05:13.947716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.939561    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:13.947716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.943001    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"]
Oct 30 18:05:13.963188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.957504    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt54t\" (UniqueName: \"kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t\") pod \"payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:13.993494 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:13.993459    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main" containerID="cri-o://e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c" gracePeriod=45
Oct 30 18:05:14.026941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.026915    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:14.117836 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.117797    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpgq\" (UniqueName: \"kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq\") pod \"payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.118103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.118081    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.219245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.219158    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpgq\" (UniqueName: \"kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq\") pod \"payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.219478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.219455    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.220463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.220431    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.234983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.234955    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:14.244879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.244537    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpgq\" (UniqueName: \"kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq\") pod \"payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.274219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.267338    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.326795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.326505    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 18:05:14.352097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.352069    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"]
Oct 30 18:05:14.391148 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.391027    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db1ff6e-79b1-4ef4-83bb-716435f6f1eb" path="/var/lib/kubelet/pods/4db1ff6e-79b1-4ef4-83bb-716435f6f1eb/volumes"
Oct 30 18:05:14.395544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.395520    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22b1f6e-24fa-437f-809d-446fbf9f894d" path="/var/lib/kubelet/pods/f22b1f6e-24fa-437f-809d-446fbf9f894d/volumes"
Oct 30 18:05:14.524842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.524804    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir\") pod \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") "
Oct 30 18:05:14.525024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.524878    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7\" (UniqueName: \"kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7\") pod \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") "
Oct 30 18:05:14.525024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.524967    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home\") pod \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") "
Oct 30 18:05:14.525179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.525096    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm\") pod \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") "
Oct 30 18:05:14.525179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.525127    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache\") pod \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") "
Oct 30 18:05:14.525293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.525176    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir" pod "b0519c04-2d73-4f8f-bf20-a88af61fbbf4" (UID: "b0519c04-2d73-4f8f-bf20-a88af61fbbf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:14.525352 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.525321    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs\") pod \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\" (UID: \"b0519c04-2d73-4f8f-bf20-a88af61fbbf4\") "
Oct 30 18:05:14.525547 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.525482    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache" pod "b0519c04-2d73-4f8f-bf20-a88af61fbbf4" (UID: "b0519c04-2d73-4f8f-bf20-a88af61fbbf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:14.525890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.525864    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home" pod "b0519c04-2d73-4f8f-bf20-a88af61fbbf4" (UID: "b0519c04-2d73-4f8f-bf20-a88af61fbbf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:14.526049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.526021    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:14.526164 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.526052    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:14.526164 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.526069    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:14.528546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.528510    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm" pod "b0519c04-2d73-4f8f-bf20-a88af61fbbf4" (UID: "b0519c04-2d73-4f8f-bf20-a88af61fbbf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:14.528711 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.528622    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs" pod "b0519c04-2d73-4f8f-bf20-a88af61fbbf4" (UID: "b0519c04-2d73-4f8f-bf20-a88af61fbbf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:14.528966 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.528939    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7" pod "b0519c04-2d73-4f8f-bf20-a88af61fbbf4" (UID: "b0519c04-2d73-4f8f-bf20-a88af61fbbf4"). InnerVolumeSpecName "kube-api-access-7lzg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:14.586070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.586047    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"]
Oct 30 18:05:14.587523 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:14.587494    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f97ec5d_56f2_4d07_b6a8_9c589e434ef0.slice/crio-06e0c45e4b2f4500d06bd07f2d50319ac028d6c4f53326279c56bc7b32ecd5b9 WatchSource:0}: Error finding container 06e0c45e4b2f4500d06bd07f2d50319ac028d6c4f53326279c56bc7b32ecd5b9: Status 404 returned error can't find the container with id 06e0c45e4b2f4500d06bd07f2d50319ac028d6c4f53326279c56bc7b32ecd5b9
Oct 30 18:05:14.609706 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.609675    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"]
Oct 30 18:05:14.610598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.610576    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main"
Oct 30 18:05:14.610598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.610596    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main"
Oct 30 18:05:14.610851 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.610803    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerName="main"
Oct 30 18:05:14.615936 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.615914    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.628693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.627724    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rzn\" (UniqueName: \"kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.628693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.627804    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.628693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.627896    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.628693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.628261    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:14.628693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.628286    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzg7\" (UniqueName: \"kubernetes.io/projected/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-kube-api-access-7lzg7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:14.628693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.628300    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0519c04-2d73-4f8f-bf20-a88af61fbbf4-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:14.631248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.631224    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:14.631572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.631290    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:14.632207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.632186    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"]
Oct 30 18:05:14.746723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.723789    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"]
Oct 30 18:05:14.746723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.740100    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rzn\" (UniqueName: \"kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.746723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.740187    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.746723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.740270    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.746723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.741411    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.796186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.795982    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:14.819919 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.819813    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rzn\" (UniqueName: \"kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.820718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.820661    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd69e8fe-9z6bb\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.822108 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.822087    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:14.824261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.824236    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:14.824369 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.824263    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"]
Oct 30 18:05:14.952154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.952107    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwxz\" (UniqueName: \"kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:14.952327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.952252    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:14.952411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.952361    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:14.954595 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.954560    5534 generic.go:356] "Generic (PLEG): container finished" podID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" containerID="e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c" exitCode=0
Oct 30 18:05:14.954816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.954795    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" event={"ID":"b0519c04-2d73-4f8f-bf20-a88af61fbbf4","Type":"ContainerDied","Data":"e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c"}
Oct 30 18:05:14.954901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.954837    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj" event={"ID":"b0519c04-2d73-4f8f-bf20-a88af61fbbf4","Type":"ContainerDied","Data":"daf17196dd7930185fb66e37a72d8e3112deab0ca799d46108216c87b6bb3263"}
Oct 30 18:05:14.954901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.954859    5534 scope.go:122] "RemoveContainer" containerID="e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c"
Oct 30 18:05:14.955010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.954994    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"
Oct 30 18:05:14.958333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.957603    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4" event={"ID":"a660b3fe-af8a-4332-8c3b-32dfcec760f5","Type":"ContainerStarted","Data":"63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119"}
Oct 30 18:05:14.958333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.957676    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4" event={"ID":"a660b3fe-af8a-4332-8c3b-32dfcec760f5","Type":"ContainerStarted","Data":"b0a39f3455e5795cef69c59c8cc17310237b2882b01ca0fdc00bb564dc1fef27"}
Oct 30 18:05:14.958333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.958292    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:14.970075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.969694    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc" event={"ID":"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0","Type":"ContainerStarted","Data":"d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b"}
Oct 30 18:05:14.970075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.969752    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc" event={"ID":"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0","Type":"ContainerStarted","Data":"06e0c45e4b2f4500d06bd07f2d50319ac028d6c4f53326279c56bc7b32ecd5b9"}
Oct 30 18:05:14.970075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.970000    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:14.970497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.970446    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:14.980339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.980313    5534 scope.go:122] "RemoveContainer" containerID="e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c"
Oct 30 18:05:14.980780 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:14.980755    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c\": container with ID starting with e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c not found: ID does not exist" containerID="e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c"
Oct 30 18:05:14.980941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:14.980917    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c"} err="failed to get container status \"e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c\": rpc error: code = NotFound desc = could not find container \"e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c\": container with ID starting with e8ca718ec4d780b94bb4183c4af4314b816dd17986326715bf5eda11f3ae873c not found: ID does not exist"
Oct 30 18:05:15.021083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.020989    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"]
Oct 30 18:05:15.032942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.032867    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-23c1e8-nrpm5"]
Oct 30 18:05:15.085834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.062181    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwxz\" (UniqueName: \"kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.085834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.062335    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.085834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.062451    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.085834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.070618    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.085834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.071230    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.094798 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.094767    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwxz\" (UniqueName: \"kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz\") pod \"maas-api-revoke-sub-20bcd8e4-9s9mw\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.095268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.095210    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4" podStartSLOduration=2.095189288 podStartE2EDuration="2.095189288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:15.028065067 +0000 UTC m=+3120.790108881" watchObservedRunningTime="2026-10-30 18:05:15.095189288 +0000 UTC m=+3120.857233102"
Oct 30 18:05:15.095934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.095896    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc" podStartSLOduration=2.095887865 podStartE2EDuration="2.095887865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:15.073073913 +0000 UTC m=+3120.835117727" watchObservedRunningTime="2026-10-30 18:05:15.095887865 +0000 UTC m=+3120.857931679"
Oct 30 18:05:15.120945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.120914    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"]
Oct 30 18:05:15.124300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.124275    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/simulator-w3-306aed-kserve-55d9769d55-l9zgj"]
Oct 30 18:05:15.130865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.130841    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:15.171571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.171539    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:15.185054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.185025    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"]
Oct 30 18:05:15.271710 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:15.271491    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:05:15.487765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.487706    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:15.546548 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:15.545822    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f27c89_9fdc_46cc_8d8e_ad9663892163.slice/crio-58f60df2d7f5c40b54d763433262021058c93efc613d845832b00857bf6dd348 WatchSource:0}: Error finding container 58f60df2d7f5c40b54d763433262021058c93efc613d845832b00857bf6dd348: Status 404 returned error can't find the container with id 58f60df2d7f5c40b54d763433262021058c93efc613d845832b00857bf6dd348
Oct 30 18:05:15.546786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.546567    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"]
Oct 30 18:05:15.983161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.983133    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb" event={"ID":"38600102-28ac-406e-a8e7-2d8c08533f55","Type":"ContainerStarted","Data":"6514ed7161b8a46f3cbcccd39311190172cec4260731554a1cb01753b5e85e76"}
Oct 30 18:05:15.983678 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.983631    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb" event={"ID":"38600102-28ac-406e-a8e7-2d8c08533f55","Type":"ContainerStarted","Data":"ee55d82348221aa7681d56cc9ad431b490199ce0911aeb9c42cd5da1d1fcdd47"}
Oct 30 18:05:15.986374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.986350    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw" event={"ID":"c0f27c89-9fdc-46cc-8d8e-ad9663892163","Type":"ContainerStarted","Data":"9530580b8cac0e8d49726d8c0edf715a3b1ec73a332450efbd09e531828f584d"}
Oct 30 18:05:15.986471 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:15.986404    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw" event={"ID":"c0f27c89-9fdc-46cc-8d8e-ad9663892163","Type":"ContainerStarted","Data":"58f60df2d7f5c40b54d763433262021058c93efc613d845832b00857bf6dd348"}
Oct 30 18:05:16.026447 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:16.026337    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb" podStartSLOduration=2.026318222 podStartE2EDuration="2.026318222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:16.024202769 +0000 UTC m=+3121.786246583" watchObservedRunningTime="2026-10-30 18:05:16.026318222 +0000 UTC m=+3121.788362034"
Oct 30 18:05:16.069104 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:16.069044    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw" podStartSLOduration=2.069023867 podStartE2EDuration="2.069023867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:16.06827229 +0000 UTC m=+3121.830316118" watchObservedRunningTime="2026-10-30 18:05:16.069023867 +0000 UTC m=+3121.831067681"
Oct 30 18:05:16.373976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:16.373941    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916b961e-91f7-4fa5-8b9d-ef564a528eed" path="/var/lib/kubelet/pods/916b961e-91f7-4fa5-8b9d-ef564a528eed/volumes"
Oct 30 18:05:16.374572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:16.374556    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0519c04-2d73-4f8f-bf20-a88af61fbbf4" path="/var/lib/kubelet/pods/b0519c04-2d73-4f8f-bf20-a88af61fbbf4/volumes"
Oct 30 18:05:16.998881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:16.998771    5534 generic.go:356] "Generic (PLEG): container finished" podID="38600102-28ac-406e-a8e7-2d8c08533f55" containerID="6514ed7161b8a46f3cbcccd39311190172cec4260731554a1cb01753b5e85e76" exitCode=0
Oct 30 18:05:16.999359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:16.998880    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb" event={"ID":"38600102-28ac-406e-a8e7-2d8c08533f55","Type":"ContainerDied","Data":"6514ed7161b8a46f3cbcccd39311190172cec4260731554a1cb01753b5e85e76"}
Oct 30 18:05:17.001221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:17.001196    5534 generic.go:356] "Generic (PLEG): container finished" podID="c0f27c89-9fdc-46cc-8d8e-ad9663892163" containerID="9530580b8cac0e8d49726d8c0edf715a3b1ec73a332450efbd09e531828f584d" exitCode=0
Oct 30 18:05:17.001458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:17.001437    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw" event={"ID":"c0f27c89-9fdc-46cc-8d8e-ad9663892163","Type":"ContainerDied","Data":"9530580b8cac0e8d49726d8c0edf715a3b1ec73a332450efbd09e531828f584d"}
Oct 30 18:05:17.403311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:17.403280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:18.088096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.088056    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm" podUID="e7395335-85f2-4166-a73f-f29c5fa1e026" containerName="payload-processing" containerID="cri-o://f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f" gracePeriod=25
Oct 30 18:05:18.234752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.234725    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:18.239161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.239140    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:18.392247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.392216    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 18:05:18.415220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415186    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz\" (UniqueName: \"kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz\") pod \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") "
Oct 30 18:05:18.415391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415313    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca\") pod \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") "
Oct 30 18:05:18.415453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415396    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca\") pod \"38600102-28ac-406e-a8e7-2d8c08533f55\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") "
Oct 30 18:05:18.415517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415462    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca\") pod \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\" (UID: \"c0f27c89-9fdc-46cc-8d8e-ad9663892163\") "
Oct 30 18:05:18.415676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415632    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca\") pod \"38600102-28ac-406e-a8e7-2d8c08533f55\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") "
Oct 30 18:05:18.415766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415699    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca" pod "c0f27c89-9fdc-46cc-8d8e-ad9663892163" (UID: "c0f27c89-9fdc-46cc-8d8e-ad9663892163"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:18.415766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.415745    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca" pod "38600102-28ac-406e-a8e7-2d8c08533f55" (UID: "38600102-28ac-406e-a8e7-2d8c08533f55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:18.416105 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.416014    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn\" (UniqueName: \"kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn\") pod \"38600102-28ac-406e-a8e7-2d8c08533f55\" (UID: \"38600102-28ac-406e-a8e7-2d8c08533f55\") "
Oct 30 18:05:18.417128 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.417089    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.417341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.417245    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.417982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.417957    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca" pod "c0f27c89-9fdc-46cc-8d8e-ad9663892163" (UID: "c0f27c89-9fdc-46cc-8d8e-ad9663892163"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:18.418076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.418000    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz" pod "c0f27c89-9fdc-46cc-8d8e-ad9663892163" (UID: "c0f27c89-9fdc-46cc-8d8e-ad9663892163"). InnerVolumeSpecName "kube-api-access-nnwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:18.418215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.418200    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca" pod "38600102-28ac-406e-a8e7-2d8c08533f55" (UID: "38600102-28ac-406e-a8e7-2d8c08533f55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:18.418304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.418289    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn" pod "38600102-28ac-406e-a8e7-2d8c08533f55" (UID: "38600102-28ac-406e-a8e7-2d8c08533f55"). InnerVolumeSpecName "kube-api-access-d8rzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:18.517779 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.517749    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk\" (UniqueName: \"kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk\") pod \"e7395335-85f2-4166-a73f-f29c5fa1e026\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") "
Oct 30 18:05:18.517965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.517798    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume\") pod \"e7395335-85f2-4166-a73f-f29c5fa1e026\" (UID: \"e7395335-85f2-4166-a73f-f29c5fa1e026\") "
Oct 30 18:05:18.518212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.518187    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume" pod "e7395335-85f2-4166-a73f-f29c5fa1e026" (UID: "e7395335-85f2-4166-a73f-f29c5fa1e026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:18.518428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.518411    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38600102-28ac-406e-a8e7-2d8c08533f55-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.518500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.518435    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rzn\" (UniqueName: \"kubernetes.io/projected/38600102-28ac-406e-a8e7-2d8c08533f55-kube-api-access-d8rzn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.518500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.518452    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7395335-85f2-4166-a73f-f29c5fa1e026-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.518500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.518466    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwxz\" (UniqueName: \"kubernetes.io/projected/c0f27c89-9fdc-46cc-8d8e-ad9663892163-kube-api-access-nnwxz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.518500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.518481    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f27c89-9fdc-46cc-8d8e-ad9663892163-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:18.520162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.520137    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk" pod "e7395335-85f2-4166-a73f-f29c5fa1e026" (UID: "e7395335-85f2-4166-a73f-f29c5fa1e026"). InnerVolumeSpecName "kube-api-access-x9tkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:18.620023 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:18.619989    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9tkk\" (UniqueName: \"kubernetes.io/projected/e7395335-85f2-4166-a73f-f29c5fa1e026-kube-api-access-x9tkk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:19.019504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.019420    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"
Oct 30 18:05:19.019504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.019487    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb" event={"ID":"38600102-28ac-406e-a8e7-2d8c08533f55","Type":"ContainerDied","Data":"ee55d82348221aa7681d56cc9ad431b490199ce0911aeb9c42cd5da1d1fcdd47"}
Oct 30 18:05:19.019736 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.019568    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee55d82348221aa7681d56cc9ad431b490199ce0911aeb9c42cd5da1d1fcdd47"
Oct 30 18:05:19.021207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.021174    5534 generic.go:356] "Generic (PLEG): container finished" podID="e7395335-85f2-4166-a73f-f29c5fa1e026" containerID="f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f" exitCode=0
Oct 30 18:05:19.021361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.021227    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"
Oct 30 18:05:19.021361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.021251    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm" event={"ID":"e7395335-85f2-4166-a73f-f29c5fa1e026","Type":"ContainerDied","Data":"f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f"}
Oct 30 18:05:19.021361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.021294    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm" event={"ID":"e7395335-85f2-4166-a73f-f29c5fa1e026","Type":"ContainerDied","Data":"2573125da6fff2770733f50a0fcd8edacf38cef157ddacd10627de1f773b76d1"}
Oct 30 18:05:19.021361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.021318    5534 scope.go:122] "RemoveContainer" containerID="f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f"
Oct 30 18:05:19.022950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.022931    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"
Oct 30 18:05:19.023038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.022967    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw" event={"ID":"c0f27c89-9fdc-46cc-8d8e-ad9663892163","Type":"ContainerDied","Data":"58f60df2d7f5c40b54d763433262021058c93efc613d845832b00857bf6dd348"}
Oct 30 18:05:19.023038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.023007    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f60df2d7f5c40b54d763433262021058c93efc613d845832b00857bf6dd348"
Oct 30 18:05:19.034990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.034974    5534 scope.go:122] "RemoveContainer" containerID="f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f"
Oct 30 18:05:19.035289 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:19.035273    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f\": container with ID starting with f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f not found: ID does not exist" containerID="f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f"
Oct 30 18:05:19.035336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.035297    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f"} err="failed to get container status \"f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f\": rpc error: code = NotFound desc = could not find container \"f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f\": container with ID starting with f5fa863d748e6c71fa93dedb608c511b594f0ca8c39d4fffe35c998cad4d4a1f not found: ID does not exist"
Oct 30 18:05:19.093244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.093204    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"]
Oct 30 18:05:19.101001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.100973    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a17354-67d9b89d78-xjrhm"]
Oct 30 18:05:19.230141 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:19.230105    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:20.253763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.253734    5534 scope.go:122] "RemoveContainer" containerID="94ad815b00dc02dac698db31370be66173bb310d74832a1377ad732271b90314"
Oct 30 18:05:20.267317 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.267279    5534 scope.go:122] "RemoveContainer" containerID="8dd26c22094dca982cbd2854e2d8d8524e01253caae236bb853e13915e6c801c"
Oct 30 18:05:20.371921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.371894    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7395335-85f2-4166-a73f-f29c5fa1e026" path="/var/lib/kubelet/pods/e7395335-85f2-4166-a73f-f29c5fa1e026/volumes"
Oct 30 18:05:20.588855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.588774    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"]
Oct 30 18:05:20.589092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.589067    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" podUID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" containerName="maas-api" containerID="cri-o://0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741" gracePeriod=30
Oct 30 18:05:20.621845 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.621817    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" podUID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.252:8443/health\": dial tcp 10.133.0.252:8443: connect: connection refused"
Oct 30 18:05:20.677313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.677281    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:20.948443 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:20.948410    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 18:05:21.045164 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.045121    5534 generic.go:356] "Generic (PLEG): container finished" podID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" containerID="0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741" exitCode=0
Oct 30 18:05:21.045404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.045185    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"
Oct 30 18:05:21.045404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.045307    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" event={"ID":"cee5f338-8334-4dde-a55d-dfa15e1ea7e2","Type":"ContainerDied","Data":"0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741"}
Oct 30 18:05:21.045404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.045366    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t" event={"ID":"cee5f338-8334-4dde-a55d-dfa15e1ea7e2","Type":"ContainerDied","Data":"b358a86f0b50c8d21fe18079ffe6a4592f226f4a3d14354d5e2d7bf39cac178b"}
Oct 30 18:05:21.045569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.045407    5534 scope.go:122] "RemoveContainer" containerID="0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741"
Oct 30 18:05:21.057368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.057346    5534 scope.go:122] "RemoveContainer" containerID="0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741"
Oct 30 18:05:21.057802 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:21.057776    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741\": container with ID starting with 0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741 not found: ID does not exist" containerID="0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741"
Oct 30 18:05:21.057923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.057810    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741"} err="failed to get container status \"0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741\": rpc error: code = NotFound desc = could not find container \"0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741\": container with ID starting with 0f47fd210e804f9ba5320b8d90321fdd6d2b64bd679dae93780877710100a741 not found: ID does not exist"
Oct 30 18:05:21.082233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.082200    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca\") pod \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") "
Oct 30 18:05:21.082457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.082283    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\") pod \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") "
Oct 30 18:05:21.082457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.082356    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs\") pod \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") "
Oct 30 18:05:21.082585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.082478    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x\" (UniqueName: \"kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x\") pod \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\" (UID: \"cee5f338-8334-4dde-a55d-dfa15e1ea7e2\") "
Oct 30 18:05:21.082717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.082693    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca" pod "cee5f338-8334-4dde-a55d-dfa15e1ea7e2" (UID: "cee5f338-8334-4dde-a55d-dfa15e1ea7e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:21.083342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.083317    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:21.087823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.085186    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x" pod "cee5f338-8334-4dde-a55d-dfa15e1ea7e2" (UID: "cee5f338-8334-4dde-a55d-dfa15e1ea7e2"). InnerVolumeSpecName "kube-api-access-lhp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:21.087823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.085196    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls" pod "cee5f338-8334-4dde-a55d-dfa15e1ea7e2" (UID: "cee5f338-8334-4dde-a55d-dfa15e1ea7e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:21.087823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.085278    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs" pod "cee5f338-8334-4dde-a55d-dfa15e1ea7e2" (UID: "cee5f338-8334-4dde-a55d-dfa15e1ea7e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:21.190293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.189701    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:21.190293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.189730    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:21.190293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.189744    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp4x\" (UniqueName: \"kubernetes.io/projected/cee5f338-8334-4dde-a55d-dfa15e1ea7e2-kube-api-access-lhp4x\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:21.190293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.190153    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:21.311136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.311106    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:21.344423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.344393    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"]
Oct 30 18:05:21.359940 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.359910    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-01b925-554dm"]
Oct 30 18:05:21.407439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.407397    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"]
Oct 30 18:05:21.412874 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.412839    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-01b925-75cd65f8-cxk2t"]
Oct 30 18:05:21.454205 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.454114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:21.840366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:21.840343    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:22.313952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.313850    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:22.380576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.380496    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d885e3-b2ca-4856-9490-1253b0aa5849" path="/var/lib/kubelet/pods/27d885e3-b2ca-4856-9490-1253b0aa5849/volumes"
Oct 30 18:05:22.381485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.381452    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" path="/var/lib/kubelet/pods/cee5f338-8334-4dde-a55d-dfa15e1ea7e2/volumes"
Oct 30 18:05:22.432099 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.432067    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:22.662555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.662520    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"]
Oct 30 18:05:22.663546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.663516    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" containerID="cri-o://ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898" gracePeriod=30
Oct 30 18:05:22.715589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:22.715553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:23.535983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:23.535947    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:23.601345 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:23.601305    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:05:25.804220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:25.804184    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:26.450347 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:26.450310    5534 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:05:26.450557 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:26.450434    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:05:27.004425 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:27.004395    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:05:27.004962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:27.004488    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:05:27.708153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:27.708126    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:27.842595 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:27.842569    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:27.963941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:27.963908    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:05:28.094863 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.094831    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.094881    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.094916    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.094955    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095003    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095048    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095157    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095228    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095305    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095313    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr\") pod \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\" (UID: \"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3\") "
Oct 30 18:05:28.095360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095329    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:28.095978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095487    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:28.095978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095499    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:28.095978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.095630    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:28.096137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.096025    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.096137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.096045    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.096137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.096059    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.096137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.096073    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.096137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.096087    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.097752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.097712    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "kube-api-access-hndkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:28.097999 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.097963    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:28.098080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.098064    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:28.098124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.098064    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo" pod "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" (UID: "ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:05:28.131342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.131319    5534 generic.go:356] "Generic (PLEG): container finished" podID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerID="ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898" exitCode=0
Oct 30 18:05:28.131452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.131363    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"
Oct 30 18:05:28.131452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.131399    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" event={"ID":"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3","Type":"ContainerDied","Data":"ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898"}
Oct 30 18:05:28.131543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.131458    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z" event={"ID":"ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3","Type":"ContainerDied","Data":"3db722ba77e9871399f495018e75015dc7f329ef22cebde963ba715a1ead8c7e"}
Oct 30 18:05:28.131543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.131482    5534 scope.go:122] "RemoveContainer" containerID="ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898"
Oct 30 18:05:28.149051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.149014    5534 scope.go:122] "RemoveContainer" containerID="ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898"
Oct 30 18:05:28.149519 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:28.149488    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898\": container with ID starting with ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898 not found: ID does not exist" containerID="ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898"
Oct 30 18:05:28.149629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.149527    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898"} err="failed to get container status \"ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898\": rpc error: code = NotFound desc = could not find container \"ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898\": container with ID starting with ebeec8249ee3abdf67094030e5c697a553c17b4fb93102d08390514ed0896898 not found: ID does not exist"
Oct 30 18:05:28.185633 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.185607    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"]
Oct 30 18:05:28.190841 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.190772    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-fbl6z"]
Oct 30 18:05:28.197424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.197387    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndkr\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-kube-api-access-hndkr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.197424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.197422    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.197574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.197436    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.197574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.197449    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:28.371290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.371257    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" path="/var/lib/kubelet/pods/ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3/volumes"
Oct 30 18:05:28.401361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.401313    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:28.529244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.529163    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:28.728894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.728859    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"]
Oct 30 18:05:28.729119 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.729097    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy" containerID="cri-o://7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66" gracePeriod=30
Oct 30 18:05:28.767053 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:28.767016    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:29.222396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.222352    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:29.381832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.381799    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"]
Oct 30 18:05:29.382871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382851    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e7395335-85f2-4166-a73f-f29c5fa1e026" containerName="payload-processing"
Oct 30 18:05:29.382949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382873    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7395335-85f2-4166-a73f-f29c5fa1e026" containerName="payload-processing"
Oct 30 18:05:29.382949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382938    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c0f27c89-9fdc-46cc-8d8e-ad9663892163" containerName="revoke-keys"
Oct 30 18:05:29.382949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382947    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f27c89-9fdc-46cc-8d8e-ad9663892163" containerName="revoke-keys"
Oct 30 18:05:29.383054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382975    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy"
Oct 30 18:05:29.383054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382983    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy"
Oct 30 18:05:29.383054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.382995    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" containerName="maas-api"
Oct 30 18:05:29.383054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383003    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" containerName="maas-api"
Oct 30 18:05:29.383054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383029    5534 cpu_manager.go:414] "containerMap: removing container" podUID="38600102-28ac-406e-a8e7-2d8c08533f55" containerName="revoke-keys"
Oct 30 18:05:29.383054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383036    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38600102-28ac-406e-a8e7-2d8c08533f55" containerName="revoke-keys"
Oct 30 18:05:29.383248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383171    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1e2f63-08a6-44b7-9b08-ceb3f8d412b3" containerName="istio-proxy"
Oct 30 18:05:29.383248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383185    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee5f338-8334-4dde-a55d-dfa15e1ea7e2" containerName="maas-api"
Oct 30 18:05:29.383248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383197    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="38600102-28ac-406e-a8e7-2d8c08533f55" containerName="revoke-keys"
Oct 30 18:05:29.383248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383207    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7395335-85f2-4166-a73f-f29c5fa1e026" containerName="payload-processing"
Oct 30 18:05:29.383248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.383223    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f27c89-9fdc-46cc-8d8e-ad9663892163" containerName="revoke-keys"
Oct 30 18:05:29.387520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.387498    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.403452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.403429    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:29.405992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.405971    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"]
Oct 30 18:05:29.413250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.413227    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:29.517077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.517052    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.517245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.517223    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.517325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.517266    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2j72\" (UniqueName: \"kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.556030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.555519    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:29.625043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.623743    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.625043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.623808    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2j72\" (UniqueName: \"kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.625043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.624008    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.625355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.625307    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.628065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.628035    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.649742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.649710    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2j72\" (UniqueName: \"kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72\") pod \"maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.680191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.680163    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 18:05:29.730902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.730859    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:29.829411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.829054    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume\") pod \"ae88df68-9b94-4341-abe2-e7155ea3f29f\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") "
Oct 30 18:05:29.829411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.829256    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx\" (UniqueName: \"kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx\") pod \"ae88df68-9b94-4341-abe2-e7155ea3f29f\" (UID: \"ae88df68-9b94-4341-abe2-e7155ea3f29f\") "
Oct 30 18:05:29.830826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.830795    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume" pod "ae88df68-9b94-4341-abe2-e7155ea3f29f" (UID: "ae88df68-9b94-4341-abe2-e7155ea3f29f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:29.836065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.836034    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx" pod "ae88df68-9b94-4341-abe2-e7155ea3f29f" (UID: "ae88df68-9b94-4341-abe2-e7155ea3f29f"). InnerVolumeSpecName "kube-api-access-686cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:29.931212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.931177    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88df68-9b94-4341-abe2-e7155ea3f29f-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:29.931212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:29.931208    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686cx\" (UniqueName: \"kubernetes.io/projected/ae88df68-9b94-4341-abe2-e7155ea3f29f-kube-api-access-686cx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:30.067184 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:30.067106    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc9d7d3_475a_480f_b0d1_6e138a46d1c3.slice/crio-5bc8407b353456cf4dac95914e6746ef015304f0e0d9d23323a020a1d11e207c WatchSource:0}: Error finding container 5bc8407b353456cf4dac95914e6746ef015304f0e0d9d23323a020a1d11e207c: Status 404 returned error can't find the container with id 5bc8407b353456cf4dac95914e6746ef015304f0e0d9d23323a020a1d11e207c
Oct 30 18:05:30.067489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.067466    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"]
Oct 30 18:05:30.149413 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.149378    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb" event={"ID":"acc9d7d3-475a-480f-b0d1-6e138a46d1c3","Type":"ContainerStarted","Data":"5bc8407b353456cf4dac95914e6746ef015304f0e0d9d23323a020a1d11e207c"}
Oct 30 18:05:30.150763 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.150737    5534 generic.go:356] "Generic (PLEG): container finished" podID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerID="38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b" exitCode=0
Oct 30 18:05:30.150855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.150783    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"
Oct 30 18:05:30.150855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.150810    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" event={"ID":"ae88df68-9b94-4341-abe2-e7155ea3f29f","Type":"ContainerDied","Data":"38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b"}
Oct 30 18:05:30.150952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.150852    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q" event={"ID":"ae88df68-9b94-4341-abe2-e7155ea3f29f","Type":"ContainerDied","Data":"eb8cdd327786ec03641c9ccbb73d3aca8c24bc5bd8663d709630758429848f12"}
Oct 30 18:05:30.150952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.150881    5534 scope.go:122] "RemoveContainer" containerID="38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b"
Oct 30 18:05:30.173958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.173737    5534 scope.go:122] "RemoveContainer" containerID="38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b"
Oct 30 18:05:30.174252 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:30.174223    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b\": container with ID starting with 38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b not found: ID does not exist" containerID="38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b"
Oct 30 18:05:30.174340 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.174265    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b"} err="failed to get container status \"38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b\": rpc error: code = NotFound desc = could not find container \"38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b\": container with ID starting with 38b51bfdcaa6b3fa236b9babb7ed84e037ac53f418f6d5887596ef1bef818d2b not found: ID does not exist"
Oct 30 18:05:30.213092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.213048    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"]
Oct 30 18:05:30.219512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.219491    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-306aed-7876d8948b-69v6q"]
Oct 30 18:05:30.375350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:30.375318    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" path="/var/lib/kubelet/pods/ae88df68-9b94-4341-abe2-e7155ea3f29f/volumes"
Oct 30 18:05:31.163167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.163127    5534 generic.go:356] "Generic (PLEG): container finished" podID="acc9d7d3-475a-480f-b0d1-6e138a46d1c3" containerID="9a12576e57aedcae276536344e37427d57769cb19aa81923443d5e90bb64aa68" exitCode=0
Oct 30 18:05:31.163384 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.163286    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb" event={"ID":"acc9d7d3-475a-480f-b0d1-6e138a46d1c3","Type":"ContainerDied","Data":"9a12576e57aedcae276536344e37427d57769cb19aa81923443d5e90bb64aa68"}
Oct 30 18:05:31.251949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.251922    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:31.394554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.394519    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"]
Oct 30 18:05:31.395107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.394798    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85" podUID="3da177a2-1e9a-4c22-8fba-dc58d448815c" containerName="payload-pre-processing" containerID="cri-o://6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c" gracePeriod=30
Oct 30 18:05:31.472496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.472460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:31.598618 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.598578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:31.830385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.830361    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:05:31.906013 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.905984    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:31.960935 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.960901    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume\") pod \"3da177a2-1e9a-4c22-8fba-dc58d448815c\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") "
Oct 30 18:05:31.961114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.961008    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2\" (UniqueName: \"kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2\") pod \"3da177a2-1e9a-4c22-8fba-dc58d448815c\" (UID: \"3da177a2-1e9a-4c22-8fba-dc58d448815c\") "
Oct 30 18:05:31.961401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.961372    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume" pod "3da177a2-1e9a-4c22-8fba-dc58d448815c" (UID: "3da177a2-1e9a-4c22-8fba-dc58d448815c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:31.961562 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.961542    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da177a2-1e9a-4c22-8fba-dc58d448815c-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:31.963404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:31.963378    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2" pod "3da177a2-1e9a-4c22-8fba-dc58d448815c" (UID: "3da177a2-1e9a-4c22-8fba-dc58d448815c"). InnerVolumeSpecName "kube-api-access-jsxt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:32.024083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.024050    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:32.042437 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.042330    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"]
Oct 30 18:05:32.052581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.052543    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f10dda-tvj66"]
Oct 30 18:05:32.063328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.063301    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsxt2\" (UniqueName: \"kubernetes.io/projected/3da177a2-1e9a-4c22-8fba-dc58d448815c-kube-api-access-jsxt2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:32.138427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.138394    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:32.178048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.178011    5534 generic.go:356] "Generic (PLEG): container finished" podID="3da177a2-1e9a-4c22-8fba-dc58d448815c" containerID="6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c" exitCode=0
Oct 30 18:05:32.178240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.178073    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"
Oct 30 18:05:32.178240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.178079    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85" event={"ID":"3da177a2-1e9a-4c22-8fba-dc58d448815c","Type":"ContainerDied","Data":"6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c"}
Oct 30 18:05:32.178240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.178147    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85" event={"ID":"3da177a2-1e9a-4c22-8fba-dc58d448815c","Type":"ContainerDied","Data":"de9ab8b2692803ae047278f002d5c8eba77d3ee2c4f88ce2068dd5e6d57f7d32"}
Oct 30 18:05:32.178240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.178181    5534 scope.go:122] "RemoveContainer" containerID="6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c"
Oct 30 18:05:32.195719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.195670    5534 scope.go:122] "RemoveContainer" containerID="6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c"
Oct 30 18:05:32.196040 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:32.196018    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c\": container with ID starting with 6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c not found: ID does not exist" containerID="6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c"
Oct 30 18:05:32.196133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.196047    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c"} err="failed to get container status \"6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c\": rpc error: code = NotFound desc = could not find container \"6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c\": container with ID starting with 6cc845042cbb52480ab2e34a659bcfabd879a686f425d77ee5dbd85dec2d0e3c not found: ID does not exist"
Oct 30 18:05:32.232909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.232882    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"]
Oct 30 18:05:32.253656 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.253607    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10dda-74f6b94779-lqd85"]
Oct 30 18:05:32.340575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.340550    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:32.374078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.374048    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce06dbe-7f28-4c9a-ae38-dc712c919980" path="/var/lib/kubelet/pods/1ce06dbe-7f28-4c9a-ae38-dc712c919980/volumes"
Oct 30 18:05:32.374448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.374434    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da177a2-1e9a-4c22-8fba-dc58d448815c" path="/var/lib/kubelet/pods/3da177a2-1e9a-4c22-8fba-dc58d448815c/volumes"
Oct 30 18:05:32.469049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.469004    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca\") pod \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") "
Oct 30 18:05:32.469571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.469290    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72\" (UniqueName: \"kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72\") pod \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") "
Oct 30 18:05:32.469571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.469323    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca" pod "acc9d7d3-475a-480f-b0d1-6e138a46d1c3" (UID: "acc9d7d3-475a-480f-b0d1-6e138a46d1c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:32.469571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.469363    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca\") pod \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\" (UID: \"acc9d7d3-475a-480f-b0d1-6e138a46d1c3\") "
Oct 30 18:05:32.470657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.470342    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:32.471785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.471733    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca" pod "acc9d7d3-475a-480f-b0d1-6e138a46d1c3" (UID: "acc9d7d3-475a-480f-b0d1-6e138a46d1c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:32.472081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.472055    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72" pod "acc9d7d3-475a-480f-b0d1-6e138a46d1c3" (UID: "acc9d7d3-475a-480f-b0d1-6e138a46d1c3"). InnerVolumeSpecName "kube-api-access-c2j72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:32.537171 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.537137    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:32.572490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.572395    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2j72\" (UniqueName: \"kubernetes.io/projected/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-kube-api-access-c2j72\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:32.572490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:32.572435    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc9d7d3-475a-480f-b0d1-6e138a46d1c3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:33.193690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.193619    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb" event={"ID":"acc9d7d3-475a-480f-b0d1-6e138a46d1c3","Type":"ContainerDied","Data":"5bc8407b353456cf4dac95914e6746ef015304f0e0d9d23323a020a1d11e207c"}
Oct 30 18:05:33.193871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.193747    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bc8407b353456cf4dac95914e6746ef015304f0e0d9d23323a020a1d11e207c"
Oct 30 18:05:33.193871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.193817    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"
Oct 30 18:05:33.198284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.198245    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"]
Oct 30 18:05:33.199224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199203    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerName="payload-processing"
Oct 30 18:05:33.199311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199227    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerName="payload-processing"
Oct 30 18:05:33.199311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199256    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3da177a2-1e9a-4c22-8fba-dc58d448815c" containerName="payload-pre-processing"
Oct 30 18:05:33.199311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199265    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da177a2-1e9a-4c22-8fba-dc58d448815c" containerName="payload-pre-processing"
Oct 30 18:05:33.199436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199332    5534 cpu_manager.go:414] "containerMap: removing container" podUID="acc9d7d3-475a-480f-b0d1-6e138a46d1c3" containerName="revoke-keys"
Oct 30 18:05:33.199436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199340    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc9d7d3-475a-480f-b0d1-6e138a46d1c3" containerName="revoke-keys"
Oct 30 18:05:33.199549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199521    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc9d7d3-475a-480f-b0d1-6e138a46d1c3" containerName="revoke-keys"
Oct 30 18:05:33.199549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199544    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae88df68-9b94-4341-abe2-e7155ea3f29f" containerName="payload-processing"
Oct 30 18:05:33.199697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.199556    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da177a2-1e9a-4c22-8fba-dc58d448815c" containerName="payload-pre-processing"
Oct 30 18:05:33.204905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.204886    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.224629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.224596    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"]
Oct 30 18:05:33.231760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.231737    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:33.231885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.231755    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:33.282953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.282912    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.283156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.283053    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.283156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.283116    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.384778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.384740    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.384990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.384830    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.384990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.384876    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.385770 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.385623    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.387589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.387562    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.408597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.408570    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.518465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.518391    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:33.838918 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.838889    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"]
Oct 30 18:05:33.840111 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:33.840087    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48297df0_703f_4fc9_b82f_fb8a191587e6.slice/crio-5606c9a9ef975eee67b64fa710f09b9064f017f25d718f6be16df2a2d01956ab WatchSource:0}: Error finding container 5606c9a9ef975eee67b64fa710f09b9064f017f25d718f6be16df2a2d01956ab: Status 404 returned error can't find the container with id 5606c9a9ef975eee67b64fa710f09b9064f017f25d718f6be16df2a2d01956ab
Oct 30 18:05:33.920058 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:33.917135    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:34.080855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.080829    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 18:05:34.188438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.188405    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"]
Oct 30 18:05:34.204291 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.204260    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"]
Oct 30 18:05:34.205875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.205840    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk" event={"ID":"48297df0-703f-4fc9-b82f-fb8a191587e6","Type":"ContainerStarted","Data":"5606c9a9ef975eee67b64fa710f09b9064f017f25d718f6be16df2a2d01956ab"}
Oct 30 18:05:34.208060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.208031    5534 generic.go:356] "Generic (PLEG): container finished" podID="4ed38c99-08e0-476e-aa19-6602e5073655" containerID="7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66" exitCode=0
Oct 30 18:05:34.208214 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.208126    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" event={"ID":"4ed38c99-08e0-476e-aa19-6602e5073655","Type":"ContainerDied","Data":"7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66"}
Oct 30 18:05:34.208214 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.208171    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4" event={"ID":"4ed38c99-08e0-476e-aa19-6602e5073655","Type":"ContainerDied","Data":"120179bad04bc958f2ecde9d2abc901b40ec5fa989d6488c0112a7621c6bf3f7"}
Oct 30 18:05:34.208214 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.208201    5534 scope.go:122] "RemoveContainer" containerID="7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66"
Oct 30 18:05:34.208375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.208348    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"
Oct 30 18:05:34.219211 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219184    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219242    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219306    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219410    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219435    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219454    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219483    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219543    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.219615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.219602    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket\") pod \"4ed38c99-08e0-476e-aa19-6602e5073655\" (UID: \"4ed38c99-08e0-476e-aa19-6602e5073655\") "
Oct 30 18:05:34.220400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.220383    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:34.221028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.220928    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:34.221140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.220966    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:34.221140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.220997    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:34.221288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.221172    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:34.223422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.222748    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "kube-api-access-xdxrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:34.223625 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.223601    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:34.224051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.224026    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:34.224137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.224121    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo" pod "4ed38c99-08e0-476e-aa19-6602e5073655" (UID: "4ed38c99-08e0-476e-aa19-6602e5073655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:05:34.227289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.227243    5534 scope.go:122] "RemoveContainer" containerID="5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"
Oct 30 18:05:34.237231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.237209    5534 scope.go:122] "RemoveContainer" containerID="7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66"
Oct 30 18:05:34.237473 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:34.237457    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66\": container with ID starting with 7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66 not found: ID does not exist" containerID="7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66"
Oct 30 18:05:34.237540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.237481    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66"} err="failed to get container status \"7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66\": rpc error: code = NotFound desc = could not find container \"7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66\": container with ID starting with 7236a73268720cd32d13806152ff352a05ebb881a5a0e8590e2b27cbd7705d66 not found: ID does not exist"
Oct 30 18:05:34.237540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.237500    5534 scope.go:122] "RemoveContainer" containerID="5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"
Oct 30 18:05:34.237735 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:34.237719    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf\": container with ID starting with 5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf not found: ID does not exist" containerID="5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"
Oct 30 18:05:34.237778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.237739    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf"} err="failed to get container status \"5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf\": rpc error: code = NotFound desc = could not find container \"5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf\": container with ID starting with 5a8e7e1002b132a75d08428319bde1ea9a437e08eca7732e9c4abb3e4a9e7adf not found: ID does not exist"
Oct 30 18:05:34.320450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.320416    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:34.321329 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321289    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321329 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321321    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed38c99-08e0-476e-aa19-6602e5073655-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321338    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321354    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321370    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321383    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321397    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxrb\" (UniqueName: \"kubernetes.io/projected/4ed38c99-08e0-476e-aa19-6602e5073655-kube-api-access-xdxrb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321409    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed38c99-08e0-476e-aa19-6602e5073655-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.321512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.321422    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed38c99-08e0-476e-aa19-6602e5073655-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:34.570475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.570392    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"]
Oct 30 18:05:34.596222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:34.596188    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-9wmw4"]
Oct 30 18:05:35.220901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:35.220807    5534 generic.go:356] "Generic (PLEG): container finished" podID="48297df0-703f-4fc9-b82f-fb8a191587e6" containerID="2ef1fd9934355367f514172906ef85972e034eac6e9b172cbb4247cd0b0d3116" exitCode=0
Oct 30 18:05:35.221087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:35.220897    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk" event={"ID":"48297df0-703f-4fc9-b82f-fb8a191587e6","Type":"ContainerDied","Data":"2ef1fd9934355367f514172906ef85972e034eac6e9b172cbb4247cd0b0d3116"}
Oct 30 18:05:35.810111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:35.810077    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:35.960678 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:35.960486    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:36.097081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.097041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:36.374748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.374718    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" path="/var/lib/kubelet/pods/4ed38c99-08e0-476e-aa19-6602e5073655/volumes"
Oct 30 18:05:36.413424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.413395    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:36.555089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.555053    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca\") pod \"48297df0-703f-4fc9-b82f-fb8a191587e6\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") "
Oct 30 18:05:36.555272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.555151    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt\") pod \"48297df0-703f-4fc9-b82f-fb8a191587e6\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") "
Oct 30 18:05:36.555366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.555349    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca\") pod \"48297df0-703f-4fc9-b82f-fb8a191587e6\" (UID: \"48297df0-703f-4fc9-b82f-fb8a191587e6\") "
Oct 30 18:05:36.556397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.556362    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca" pod "48297df0-703f-4fc9-b82f-fb8a191587e6" (UID: "48297df0-703f-4fc9-b82f-fb8a191587e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:36.556589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.556566    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:36.558070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.558043    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca" pod "48297df0-703f-4fc9-b82f-fb8a191587e6" (UID: "48297df0-703f-4fc9-b82f-fb8a191587e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:36.563904 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.563880    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt" pod "48297df0-703f-4fc9-b82f-fb8a191587e6" (UID: "48297df0-703f-4fc9-b82f-fb8a191587e6"). InnerVolumeSpecName "kube-api-access-fzgtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:36.657555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.657466    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48297df0-703f-4fc9-b82f-fb8a191587e6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:36.657555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:36.657505    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/48297df0-703f-4fc9-b82f-fb8a191587e6-kube-api-access-fzgtt\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:37.139188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:37.139158    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:37.237852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:37.237821    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"
Oct 30 18:05:37.238010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:37.237845    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk" event={"ID":"48297df0-703f-4fc9-b82f-fb8a191587e6","Type":"ContainerDied","Data":"5606c9a9ef975eee67b64fa710f09b9064f017f25d718f6be16df2a2d01956ab"}
Oct 30 18:05:37.238010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:37.237903    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5606c9a9ef975eee67b64fa710f09b9064f017f25d718f6be16df2a2d01956ab"
Oct 30 18:05:37.612097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:37.612067    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:38.620962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:38.620929    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:38.911956 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:38.911870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:39.045860 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.045441    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"]
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046477    5534 cpu_manager.go:414] "containerMap: removing container" podUID="48297df0-703f-4fc9-b82f-fb8a191587e6" containerName="revoke-keys"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046502    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48297df0-703f-4fc9-b82f-fb8a191587e6" containerName="revoke-keys"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046525    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046534    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046574    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046582    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046770    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="48297df0-703f-4fc9-b82f-fb8a191587e6" containerName="revoke-keys"
Oct 30 18:05:39.047671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.046791    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy"
Oct 30 18:05:39.056943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.056895    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.073757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.073611    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"]
Oct 30 18:05:39.082665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.082345    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d323f5-openshift-default-dockercfg-np5wc\""
Oct 30 18:05:39.098334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.098309    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:39.101881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.101845    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.101964    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102129 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102109    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102151    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqv6\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102193    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102293    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102354    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102427    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.102540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.102517    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.203776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203691    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.203776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203732    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqv6\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.203776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203768    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203803    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203843    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203907    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.203936    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.204027    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.204089    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.204509    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.204899    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.204930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.204924    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.205277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.205249    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.205386 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.205360    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.207162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.207133    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.207675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.207616    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.230051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.230025    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.230256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.230235    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqv6\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6\") pod \"e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") " pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.239945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.239915    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:39.379667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.378827    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:39.414353 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.413905    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:39.757247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.757218    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"]
Oct 30 18:05:39.760268 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:39.760233    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9355dd9f_4e3b_4584_a332_068331172693.slice/crio-98fabf6e8746d86449a6714924d8d67638d8d1adc3a599aaf4b170bca41617b9 WatchSource:0}: Error finding container 98fabf6e8746d86449a6714924d8d67638d8d1adc3a599aaf4b170bca41617b9: Status 404 returned error can't find the container with id 98fabf6e8746d86449a6714924d8d67638d8d1adc3a599aaf4b170bca41617b9
Oct 30 18:05:39.764118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.764069    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:05:39.764283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.764201    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:05:39.764283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.764244    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:05:39.871355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:39.871330    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"]
Oct 30 18:05:40.096757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.096716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:40.216948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.214990    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:40.280837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.280451    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" event={"ID":"9355dd9f-4e3b-4584-a332-068331172693","Type":"ContainerStarted","Data":"67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"}
Oct 30 18:05:40.282230 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.282204    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" event={"ID":"9355dd9f-4e3b-4584-a332-068331172693","Type":"ContainerStarted","Data":"98fabf6e8746d86449a6714924d8d67638d8d1adc3a599aaf4b170bca41617b9"}
Oct 30 18:05:40.345801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.345739    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podStartSLOduration=1.345719927 podStartE2EDuration="1.345719927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:40.3407816 +0000 UTC m=+3146.102825423" watchObservedRunningTime="2026-10-30 18:05:40.345719927 +0000 UTC m=+3146.107763742"
Oct 30 18:05:40.354029 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.353954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:40.419980 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.419950    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:40.427105 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.427076    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:40.467628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.467596    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:40.932189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.932152    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"]
Oct 30 18:05:40.951025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.950003    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed38c99-08e0-476e-aa19-6602e5073655" containerName="istio-proxy"
Oct 30 18:05:40.974600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.972985    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"]
Oct 30 18:05:40.974600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.973095    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:40.987695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.987627    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:40.988279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:40.988256    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:41.085316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.085279    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knr6\" (UniqueName: \"kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.085533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.085505    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.085607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.085560    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.186497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.186415    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.186497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.186455    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.186764 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.186598    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knr6\" (UniqueName: \"kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.187220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.187197    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.189158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.189136    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.210262 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.210232    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knr6\" (UniqueName: \"kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.289477 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.289442    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:41.290438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.290415    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:05:41.290573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.290516    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:41.626186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.626153    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"]
Oct 30 18:05:41.626536 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:41.626507    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e715a0_78dc_4d01_b5bd_f7116e791ee6.slice/crio-e5613b2193e2bbefb776f5f3fe13b9366da9cc7a9265b46407b6112d0a09fe75 WatchSource:0}: Error finding container e5613b2193e2bbefb776f5f3fe13b9366da9cc7a9265b46407b6112d0a09fe75: Status 404 returned error can't find the container with id e5613b2193e2bbefb776f5f3fe13b9366da9cc7a9265b46407b6112d0a09fe75
Oct 30 18:05:41.914447 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.914359    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"]
Oct 30 18:05:41.914633 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.914598    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" podUID="639be5d9-7161-485f-9d31-d5f0714a9f21" containerName="maas-api" containerID="cri-o://2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97" gracePeriod=30
Oct 30 18:05:41.996237 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:41.996197    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:42.272588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.272559    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 18:05:42.300496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.300462    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht" event={"ID":"f6e715a0-78dc-4d01-b5bd-f7116e791ee6","Type":"ContainerStarted","Data":"40f4d7cdd6dbe0e3a0476068e4590c67fbaab52c9ba1edddf7d7678b0d022859"}
Oct 30 18:05:42.300732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.300529    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht" event={"ID":"f6e715a0-78dc-4d01-b5bd-f7116e791ee6","Type":"ContainerStarted","Data":"e5613b2193e2bbefb776f5f3fe13b9366da9cc7a9265b46407b6112d0a09fe75"}
Oct 30 18:05:42.302137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.302107    5534 generic.go:356] "Generic (PLEG): container finished" podID="639be5d9-7161-485f-9d31-d5f0714a9f21" containerID="2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97" exitCode=0
Oct 30 18:05:42.302284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.302137    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" event={"ID":"639be5d9-7161-485f-9d31-d5f0714a9f21","Type":"ContainerDied","Data":"2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97"}
Oct 30 18:05:42.302284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.302195    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9" event={"ID":"639be5d9-7161-485f-9d31-d5f0714a9f21","Type":"ContainerDied","Data":"3c4fc4729888a6f33d620f62fa2ddc257b9a615d78527497a42c3e50fb7e1445"}
Oct 30 18:05:42.302284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.302225    5534 scope.go:122] "RemoveContainer" containerID="2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97"
Oct 30 18:05:42.302284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.302146    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"
Oct 30 18:05:42.313811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.313790    5534 scope.go:122] "RemoveContainer" containerID="2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97"
Oct 30 18:05:42.314101 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:42.314082    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97\": container with ID starting with 2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97 not found: ID does not exist" containerID="2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97"
Oct 30 18:05:42.314199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.314108    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97"} err="failed to get container status \"2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97\": rpc error: code = NotFound desc = could not find container \"2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97\": container with ID starting with 2c4845926f279d0dd0c6dc7ecc05335cacca57064ae5443550e4bcf9de154e97 not found: ID does not exist"
Oct 30 18:05:42.328336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.328306    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca\") pod \"639be5d9-7161-485f-9d31-d5f0714a9f21\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") "
Oct 30 18:05:42.328480 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.328361    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\") pod \"639be5d9-7161-485f-9d31-d5f0714a9f21\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") "
Oct 30 18:05:42.328588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.328569    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs\") pod \"639be5d9-7161-485f-9d31-d5f0714a9f21\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") "
Oct 30 18:05:42.328669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.328636    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9\" (UniqueName: \"kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9\") pod \"639be5d9-7161-485f-9d31-d5f0714a9f21\" (UID: \"639be5d9-7161-485f-9d31-d5f0714a9f21\") "
Oct 30 18:05:42.328780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.328753    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca" pod "639be5d9-7161-485f-9d31-d5f0714a9f21" (UID: "639be5d9-7161-485f-9d31-d5f0714a9f21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:42.330539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.329268    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639be5d9-7161-485f-9d31-d5f0714a9f21-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:42.388140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.388062    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs" pod "639be5d9-7161-485f-9d31-d5f0714a9f21" (UID: "639be5d9-7161-485f-9d31-d5f0714a9f21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:42.447433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.442431    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:42.447433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.446295    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls" pod "639be5d9-7161-485f-9d31-d5f0714a9f21" (UID: "639be5d9-7161-485f-9d31-d5f0714a9f21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:42.480364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.469107    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9" pod "639be5d9-7161-485f-9d31-d5f0714a9f21" (UID: "639be5d9-7161-485f-9d31-d5f0714a9f21"). InnerVolumeSpecName "kube-api-access-cgbr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:42.543461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.543431    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639be5d9-7161-485f-9d31-d5f0714a9f21-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:42.543461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.543460    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgbr9\" (UniqueName: \"kubernetes.io/projected/639be5d9-7161-485f-9d31-d5f0714a9f21-kube-api-access-cgbr9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:42.641659 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.641551    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht" podStartSLOduration=2.641535193 podStartE2EDuration="2.641535193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:42.46890638 +0000 UTC m=+3148.230950194" watchObservedRunningTime="2026-10-30 18:05:42.641535193 +0000 UTC m=+3148.403579006"
Oct 30 18:05:42.645078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.645058    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"]
Oct 30 18:05:42.651923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.651900    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a17354-5dd7d4969d-r4gl9"]
Oct 30 18:05:42.753499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.753419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:42.865610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.865580    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:42.883533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.883502    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"]
Oct 30 18:05:42.887380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.887356    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a17354-b9vjx"]
Oct 30 18:05:42.976129 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:42.976096    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:43.115769 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.115740    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:43.266863 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.266831    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:43.311875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.311839    5534 generic.go:356] "Generic (PLEG): container finished" podID="f6e715a0-78dc-4d01-b5bd-f7116e791ee6" containerID="40f4d7cdd6dbe0e3a0476068e4590c67fbaab52c9ba1edddf7d7678b0d022859" exitCode=0
Oct 30 18:05:43.312057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.311927    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht" event={"ID":"f6e715a0-78dc-4d01-b5bd-f7116e791ee6","Type":"ContainerDied","Data":"40f4d7cdd6dbe0e3a0476068e4590c67fbaab52c9ba1edddf7d7678b0d022859"}
Oct 30 18:05:43.401398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.401322    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:43.697908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.697821    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"]
Oct 30 18:05:43.809434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.809270    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:43.926427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:43.926397    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:44.436668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.433869    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639be5d9-7161-485f-9d31-d5f0714a9f21" path="/var/lib/kubelet/pods/639be5d9-7161-485f-9d31-d5f0714a9f21/volumes"
Oct 30 18:05:44.436668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.434563    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c091550f-1b3e-4c55-b006-21e6adaa8fc8" path="/var/lib/kubelet/pods/c091550f-1b3e-4c55-b006-21e6adaa8fc8/volumes"
Oct 30 18:05:44.569557 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.569532    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:44.590781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.590756    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:44.680992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.680963    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:44.687314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.687290    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6\" (UniqueName: \"kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6\") pod \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") "
Oct 30 18:05:44.687445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.687381    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca\") pod \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") "
Oct 30 18:05:44.687445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.687426    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca\") pod \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\" (UID: \"f6e715a0-78dc-4d01-b5bd-f7116e791ee6\") "
Oct 30 18:05:44.687798 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.687774    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca" pod "f6e715a0-78dc-4d01-b5bd-f7116e791ee6" (UID: "f6e715a0-78dc-4d01-b5bd-f7116e791ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:44.688050 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.688021    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:44.689791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.689762    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca" pod "f6e715a0-78dc-4d01-b5bd-f7116e791ee6" (UID: "f6e715a0-78dc-4d01-b5bd-f7116e791ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:44.689933 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.689909    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6" pod "f6e715a0-78dc-4d01-b5bd-f7116e791ee6" (UID: "f6e715a0-78dc-4d01-b5bd-f7116e791ee6"). InnerVolumeSpecName "kube-api-access-6knr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:44.789754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.789716    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knr6\" (UniqueName: \"kubernetes.io/projected/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-kube-api-access-6knr6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:44.789754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.789746    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e715a0-78dc-4d01-b5bd-f7116e791ee6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:44.846479 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.846397    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:44.872191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:44.872134    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerName="payload-processing" containerID="cri-o://3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84" gracePeriod=25
Oct 30 18:05:45.021829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.021709    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:45.109569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.109489    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"]
Oct 30 18:05:45.110204 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.110188    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f6e715a0-78dc-4d01-b5bd-f7116e791ee6" containerName="revoke-keys"
Oct 30 18:05:45.110268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.110216    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e715a0-78dc-4d01-b5bd-f7116e791ee6" containerName="revoke-keys"
Oct 30 18:05:45.110268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.110234    5534 cpu_manager.go:414] "containerMap: removing container" podUID="639be5d9-7161-485f-9d31-d5f0714a9f21" containerName="maas-api"
Oct 30 18:05:45.110268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.110239    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639be5d9-7161-485f-9d31-d5f0714a9f21" containerName="maas-api"
Oct 30 18:05:45.110365 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.110360    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e715a0-78dc-4d01-b5bd-f7116e791ee6" containerName="revoke-keys"
Oct 30 18:05:45.110402 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.110368    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="639be5d9-7161-485f-9d31-d5f0714a9f21" containerName="maas-api"
Oct 30 18:05:45.113883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.113862    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.131023 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.130971    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d323f5\""
Oct 30 18:05:45.137033 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.137011    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"]
Oct 30 18:05:45.141413 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.141393    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"]
Oct 30 18:05:45.141554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.141540    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.164600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.164546    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:45.173719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.173537    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"]
Oct 30 18:05:45.193769 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.193744    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d323f5-dockercfg-lbmwv\""
Oct 30 18:05:45.200594 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.200569    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts2f\" (UniqueName: \"kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.200862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.200826    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.263570 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:45.263461    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:05:45.314473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.312682    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kk8\" (UniqueName: \"kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8\") pod \"payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.314473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.312859    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts2f\" (UniqueName: \"kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.314473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.312965    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.314473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.313067    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.314473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.314087    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.335669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.334205    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"
Oct 30 18:05:45.335669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.334253    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht" event={"ID":"f6e715a0-78dc-4d01-b5bd-f7116e791ee6","Type":"ContainerDied","Data":"e5613b2193e2bbefb776f5f3fe13b9366da9cc7a9265b46407b6112d0a09fe75"}
Oct 30 18:05:45.335669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.334297    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5613b2193e2bbefb776f5f3fe13b9366da9cc7a9265b46407b6112d0a09fe75"
Oct 30 18:05:45.358667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.341569    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts2f\" (UniqueName: \"kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.414255 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.414167    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kk8\" (UniqueName: \"kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8\") pod \"payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.414421 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.414343    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.415163 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.415137    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.430399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.430374    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:45.450052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.450019    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kk8\" (UniqueName: \"kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8\") pod \"payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.472260 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.472229    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:45.747449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.747408    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"]
Oct 30 18:05:45.748691 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:45.748630    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf26a2c_10f9_4016_972c_3e7708beb128.slice/crio-88891ec3348aec677ff6c0beee189af35bb2fd6285abf5923878bb5d7ceb24b4 WatchSource:0}: Error finding container 88891ec3348aec677ff6c0beee189af35bb2fd6285abf5923878bb5d7ceb24b4: Status 404 returned error can't find the container with id 88891ec3348aec677ff6c0beee189af35bb2fd6285abf5923878bb5d7ceb24b4
Oct 30 18:05:45.804813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:45.804666    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"]
Oct 30 18:05:45.806599 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:45.806571    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db330d8_6e15_4eda_8832_346989df4ecf.slice/crio-fd9f63e55f62276ce6ac4dda6d109a42c5ceb34a73557c7eda674a09054a01f9 WatchSource:0}: Error finding container fd9f63e55f62276ce6ac4dda6d109a42c5ceb34a73557c7eda674a09054a01f9: Status 404 returned error can't find the container with id fd9f63e55f62276ce6ac4dda6d109a42c5ceb34a73557c7eda674a09054a01f9
Oct 30 18:05:46.344341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.344303    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z" event={"ID":"7db330d8-6e15-4eda-8832-346989df4ecf","Type":"ContainerStarted","Data":"6ef3e50532f80933bba36b9f1d2f32df50f1709893506bcb16979583bf82bf12"}
Oct 30 18:05:46.344542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.344361    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z" event={"ID":"7db330d8-6e15-4eda-8832-346989df4ecf","Type":"ContainerStarted","Data":"fd9f63e55f62276ce6ac4dda6d109a42c5ceb34a73557c7eda674a09054a01f9"}
Oct 30 18:05:46.344542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.344393    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:46.346250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.346223    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr" event={"ID":"cbf26a2c-10f9-4016-972c-3e7708beb128","Type":"ContainerStarted","Data":"b46a7dab69153bd851a2c4f3f6665a75cbae5b797c5df3c500e088014ca87a68"}
Oct 30 18:05:46.346358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.346265    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr" event={"ID":"cbf26a2c-10f9-4016-972c-3e7708beb128","Type":"ContainerStarted","Data":"88891ec3348aec677ff6c0beee189af35bb2fd6285abf5923878bb5d7ceb24b4"}
Oct 30 18:05:46.346358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.346294    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:46.377923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.377895    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"]
Oct 30 18:05:46.382197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.382173    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.401984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.400022    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:46.411950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.411924    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:46.413183 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.413138    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z" podStartSLOduration=1.413123002 podStartE2EDuration="1.413123002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:46.399970258 +0000 UTC m=+3152.162014069" watchObservedRunningTime="2026-10-30 18:05:46.413123002 +0000 UTC m=+3152.175166818"
Oct 30 18:05:46.421718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.421675    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"]
Oct 30 18:05:46.429864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.426237    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"]
Oct 30 18:05:46.429864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.426333    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.434340 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.433915    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"]
Oct 30 18:05:46.497911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.497855    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr" podStartSLOduration=1.497837279 podStartE2EDuration="1.497837279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:46.495729631 +0000 UTC m=+3152.257773446" watchObservedRunningTime="2026-10-30 18:05:46.497837279 +0000 UTC m=+3152.259881093"
Oct 30 18:05:46.529427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.529400    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.529597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.529539    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.529597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.529569    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.529786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.529765    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8bw\" (UniqueName: \"kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.529931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.529914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7stf\" (UniqueName: \"kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.530003 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.529989    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.632063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.631978    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.632247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.632112    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.632247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.632141    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.632247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.632183    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8bw\" (UniqueName: \"kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.632247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.632229    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7stf\" (UniqueName: \"kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.632399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.632294    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.632954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.632921    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.633119 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.633099    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.634667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.634632    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.635316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.635292    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.669814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.669787    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8bw\" (UniqueName: \"kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw\") pod \"maas-api-revoke-sub-37c35a00-g2kc8\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.669923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.669861    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7stf\" (UniqueName: \"kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf\") pod \"maas-api-revoke-sub-018aff3d-gzdtd\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.698228 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.698200    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:46.745724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.745691    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:46.807003 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:46.806821    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:47.012537 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:47.012496    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63b2055_6998_4ada_8b98_6986ef43ecc0.slice/crio-02c1ccc537c982a83edf4953c2bce2ef7121f366222a31b4a34ec658bed5a5ad WatchSource:0}: Error finding container 02c1ccc537c982a83edf4953c2bce2ef7121f366222a31b4a34ec658bed5a5ad: Status 404 returned error can't find the container with id 02c1ccc537c982a83edf4953c2bce2ef7121f366222a31b4a34ec658bed5a5ad
Oct 30 18:05:47.013201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:47.013177    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"]
Oct 30 18:05:47.065602 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:47.065565    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"]
Oct 30 18:05:47.065850 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:47.065828    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff702ec3_0e0d_4cbc_b57e_e5f10437634e.slice/crio-c64c20e0400dd02a1d708c1ec00cf0289ac35763f3971d10dc1c2a1923a71146 WatchSource:0}: Error finding container c64c20e0400dd02a1d708c1ec00cf0289ac35763f3971d10dc1c2a1923a71146: Status 404 returned error can't find the container with id c64c20e0400dd02a1d708c1ec00cf0289ac35763f3971d10dc1c2a1923a71146
Oct 30 18:05:47.364167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:47.364129    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd" event={"ID":"f63b2055-6998-4ada-8b98-6986ef43ecc0","Type":"ContainerStarted","Data":"9acc881e506c4a449db4643a1fe6e5f1fc55c1a27b52d9f7a5a3d0575416ee48"}
Oct 30 18:05:47.364501 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:47.364479    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd" event={"ID":"f63b2055-6998-4ada-8b98-6986ef43ecc0","Type":"ContainerStarted","Data":"02c1ccc537c982a83edf4953c2bce2ef7121f366222a31b4a34ec658bed5a5ad"}
Oct 30 18:05:47.371039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:47.370138    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8" event={"ID":"ff702ec3-0e0d-4cbc-b57e-e5f10437634e","Type":"ContainerStarted","Data":"c64c20e0400dd02a1d708c1ec00cf0289ac35763f3971d10dc1c2a1923a71146"}
Oct 30 18:05:47.411465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:47.411412    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd" podStartSLOduration=1.411398722 podStartE2EDuration="1.411398722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:47.408082416 +0000 UTC m=+3153.170126230" watchObservedRunningTime="2026-10-30 18:05:47.411398722 +0000 UTC m=+3153.173442534"
Oct 30 18:05:48.378765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:48.378729    5534 generic.go:356] "Generic (PLEG): container finished" podID="f63b2055-6998-4ada-8b98-6986ef43ecc0" containerID="9acc881e506c4a449db4643a1fe6e5f1fc55c1a27b52d9f7a5a3d0575416ee48" exitCode=0
Oct 30 18:05:48.379232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:48.378811    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd" event={"ID":"f63b2055-6998-4ada-8b98-6986ef43ecc0","Type":"ContainerDied","Data":"9acc881e506c4a449db4643a1fe6e5f1fc55c1a27b52d9f7a5a3d0575416ee48"}
Oct 30 18:05:48.381049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:48.381019    5534 generic.go:356] "Generic (PLEG): container finished" podID="ff702ec3-0e0d-4cbc-b57e-e5f10437634e" containerID="f381e03c0525d058aba91b7e7d4a3c7bff02ad04eadd47110fa4e1a8f298b1cb" exitCode=0
Oct 30 18:05:48.381201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:48.381134    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8" event={"ID":"ff702ec3-0e0d-4cbc-b57e-e5f10437634e","Type":"ContainerDied","Data":"f381e03c0525d058aba91b7e7d4a3c7bff02ad04eadd47110fa4e1a8f298b1cb"}
Oct 30 18:05:48.649946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:48.649864    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:48.698674 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:48.698135    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerName="payload-pre-processing" containerID="cri-o://b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d" gracePeriod=25
Oct 30 18:05:49.217590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.217553    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main" containerID="cri-o://ed4209b545eb1c0749623bf68087118239db59ae85df7a4ff51138da71f2a785" gracePeriod=45
Oct 30 18:05:49.231080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.231049    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main" containerID="cri-o://c96d8aa4bd95fc3a9d1f5b1cab0b45b940d25ab77cb756ead43517eb7f184dc5" gracePeriod=45
Oct 30 18:05:49.392193 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.392155    5534 generic.go:356] "Generic (PLEG): container finished" podID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerID="ed4209b545eb1c0749623bf68087118239db59ae85df7a4ff51138da71f2a785" exitCode=0
Oct 30 18:05:49.392693 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.392255    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" event={"ID":"a53ac57c-cc71-4a25-8d14-97b27b9dedd6","Type":"ContainerDied","Data":"ed4209b545eb1c0749623bf68087118239db59ae85df7a4ff51138da71f2a785"}
Oct 30 18:05:49.393830 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.393805    5534 generic.go:356] "Generic (PLEG): container finished" podID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerID="c96d8aa4bd95fc3a9d1f5b1cab0b45b940d25ab77cb756ead43517eb7f184dc5" exitCode=0
Oct 30 18:05:49.394022 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.393986    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" event={"ID":"526a3a13-0745-46f4-a20b-44ee8bceb95b","Type":"ContainerDied","Data":"c96d8aa4bd95fc3a9d1f5b1cab0b45b940d25ab77cb756ead43517eb7f184dc5"}
Oct 30 18:05:49.576087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.574233    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 18:05:49.577415 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.577349    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 18:05:49.595814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.595792    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:49.664308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.663293    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:49.683764 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.683712    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home\") pod \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") "
Oct 30 18:05:49.684150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684123    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home" pod "a53ac57c-cc71-4a25-8d14-97b27b9dedd6" (UID: "a53ac57c-cc71-4a25-8d14-97b27b9dedd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.684275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684129    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca\") pod \"f63b2055-6998-4ada-8b98-6986ef43ecc0\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") "
Oct 30 18:05:49.684275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684208    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir\") pod \"526a3a13-0745-46f4-a20b-44ee8bceb95b\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") "
Oct 30 18:05:49.684275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684248    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca\") pod \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") "
Oct 30 18:05:49.684457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684288    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm\") pod \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") "
Oct 30 18:05:49.684457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684326    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir\") pod \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") "
Oct 30 18:05:49.684457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684361    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home\") pod \"526a3a13-0745-46f4-a20b-44ee8bceb95b\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") "
Oct 30 18:05:49.684457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684386    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf\" (UniqueName: \"kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf\") pod \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") "
Oct 30 18:05:49.684457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684437    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw\" (UniqueName: \"kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw\") pod \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") "
Oct 30 18:05:49.684457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684458    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs\") pod \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684477    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache\") pod \"526a3a13-0745-46f4-a20b-44ee8bceb95b\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684499    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca\") pod \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\" (UID: \"ff702ec3-0e0d-4cbc-b57e-e5f10437634e\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684525    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf\" (UniqueName: \"kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf\") pod \"f63b2055-6998-4ada-8b98-6986ef43ecc0\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684556    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm\") pod \"526a3a13-0745-46f4-a20b-44ee8bceb95b\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684583    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca\") pod \"f63b2055-6998-4ada-8b98-6986ef43ecc0\" (UID: \"f63b2055-6998-4ada-8b98-6986ef43ecc0\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684635    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6\" (UniqueName: \"kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6\") pod \"526a3a13-0745-46f4-a20b-44ee8bceb95b\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684746    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs\" (UniqueName: \"kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs\") pod \"526a3a13-0745-46f4-a20b-44ee8bceb95b\" (UID: \"526a3a13-0745-46f4-a20b-44ee8bceb95b\") "
Oct 30 18:05:49.684852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.684792    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache\") pod \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\" (UID: \"a53ac57c-cc71-4a25-8d14-97b27b9dedd6\") "
Oct 30 18:05:49.685310 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.685203    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.685400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.685375    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache" pod "a53ac57c-cc71-4a25-8d14-97b27b9dedd6" (UID: "a53ac57c-cc71-4a25-8d14-97b27b9dedd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.685680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.685525    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir" pod "526a3a13-0745-46f4-a20b-44ee8bceb95b" (UID: "526a3a13-0745-46f4-a20b-44ee8bceb95b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.685861 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.685729    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache" pod "526a3a13-0745-46f4-a20b-44ee8bceb95b" (UID: "526a3a13-0745-46f4-a20b-44ee8bceb95b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.685861 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.685745    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir" pod "a53ac57c-cc71-4a25-8d14-97b27b9dedd6" (UID: "a53ac57c-cc71-4a25-8d14-97b27b9dedd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.686088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.686065    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca" pod "ff702ec3-0e0d-4cbc-b57e-e5f10437634e" (UID: "ff702ec3-0e0d-4cbc-b57e-e5f10437634e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:49.686175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.686089    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home" pod "526a3a13-0745-46f4-a20b-44ee8bceb95b" (UID: "526a3a13-0745-46f4-a20b-44ee8bceb95b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.686881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.686858    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca" pod "f63b2055-6998-4ada-8b98-6986ef43ecc0" (UID: "f63b2055-6998-4ada-8b98-6986ef43ecc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:49.694810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.694705    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca" pod "ff702ec3-0e0d-4cbc-b57e-e5f10437634e" (UID: "ff702ec3-0e0d-4cbc-b57e-e5f10437634e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:49.694810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.694781    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm" pod "a53ac57c-cc71-4a25-8d14-97b27b9dedd6" (UID: "a53ac57c-cc71-4a25-8d14-97b27b9dedd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.695011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.694992    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf" pod "f63b2055-6998-4ada-8b98-6986ef43ecc0" (UID: "f63b2055-6998-4ada-8b98-6986ef43ecc0"). InnerVolumeSpecName "kube-api-access-j7stf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:49.695204 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.695165    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw" pod "ff702ec3-0e0d-4cbc-b57e-e5f10437634e" (UID: "ff702ec3-0e0d-4cbc-b57e-e5f10437634e"). InnerVolumeSpecName "kube-api-access-4j8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:49.695381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.695351    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs" pod "526a3a13-0745-46f4-a20b-44ee8bceb95b" (UID: "526a3a13-0745-46f4-a20b-44ee8bceb95b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:49.695597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.695546    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm" pod "526a3a13-0745-46f4-a20b-44ee8bceb95b" (UID: "526a3a13-0745-46f4-a20b-44ee8bceb95b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:49.695597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.695568    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6" pod "526a3a13-0745-46f4-a20b-44ee8bceb95b" (UID: "526a3a13-0745-46f4-a20b-44ee8bceb95b"). InnerVolumeSpecName "kube-api-access-nd6b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:49.695891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.695867    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca" pod "f63b2055-6998-4ada-8b98-6986ef43ecc0" (UID: "f63b2055-6998-4ada-8b98-6986ef43ecc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:49.697712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.697670    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs" pod "a53ac57c-cc71-4a25-8d14-97b27b9dedd6" (UID: "a53ac57c-cc71-4a25-8d14-97b27b9dedd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:49.698006 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.697922    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf" pod "a53ac57c-cc71-4a25-8d14-97b27b9dedd6" (UID: "a53ac57c-cc71-4a25-8d14-97b27b9dedd6"). InnerVolumeSpecName "kube-api-access-44zzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786738    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786795    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786811    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786829    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786843    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786855    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786867    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786880    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44zzf\" (UniqueName: \"kubernetes.io/projected/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-kube-api-access-44zzf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786895    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8bw\" (UniqueName: \"kubernetes.io/projected/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-kube-api-access-4j8bw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786911    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53ac57c-cc71-4a25-8d14-97b27b9dedd6-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786925    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786938    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff702ec3-0e0d-4cbc-b57e-e5f10437634e-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786952    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7stf\" (UniqueName: \"kubernetes.io/projected/f63b2055-6998-4ada-8b98-6986ef43ecc0-kube-api-access-j7stf\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786964    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/526a3a13-0745-46f4-a20b-44ee8bceb95b-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786978    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63b2055-6998-4ada-8b98-6986ef43ecc0-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.786994    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd6b6\" (UniqueName: \"kubernetes.io/projected/526a3a13-0745-46f4-a20b-44ee8bceb95b-kube-api-access-nd6b6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:49.787020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:49.787007    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/526a3a13-0745-46f4-a20b-44ee8bceb95b-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:50.133198 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.133163    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:50.317946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.317910    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:50.404782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.404752    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"
Oct 30 18:05:50.404782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.404772    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8" event={"ID":"a53ac57c-cc71-4a25-8d14-97b27b9dedd6","Type":"ContainerDied","Data":"03ab51f877174d21095bda7b3d7fb560a0a51ff669e96ccfe76f3dfb8dad4496"}
Oct 30 18:05:50.405305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.404824    5534 scope.go:122] "RemoveContainer" containerID="ed4209b545eb1c0749623bf68087118239db59ae85df7a4ff51138da71f2a785"
Oct 30 18:05:50.406687 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.406626    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5" event={"ID":"526a3a13-0745-46f4-a20b-44ee8bceb95b","Type":"ContainerDied","Data":"8da3f1e7305cc9833d5538fc9b36277185a24e94d05d569eef7e932fe289a304"}
Oct 30 18:05:50.406877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.406631    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"
Oct 30 18:05:50.409695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.409633    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"
Oct 30 18:05:50.409854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.409662    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd" event={"ID":"f63b2055-6998-4ada-8b98-6986ef43ecc0","Type":"ContainerDied","Data":"02c1ccc537c982a83edf4953c2bce2ef7121f366222a31b4a34ec658bed5a5ad"}
Oct 30 18:05:50.409854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.409799    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c1ccc537c982a83edf4953c2bce2ef7121f366222a31b4a34ec658bed5a5ad"
Oct 30 18:05:50.412740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.412718    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"
Oct 30 18:05:50.412869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.412760    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8" event={"ID":"ff702ec3-0e0d-4cbc-b57e-e5f10437634e","Type":"ContainerDied","Data":"c64c20e0400dd02a1d708c1ec00cf0289ac35763f3971d10dc1c2a1923a71146"}
Oct 30 18:05:50.412869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.412834    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64c20e0400dd02a1d708c1ec00cf0289ac35763f3971d10dc1c2a1923a71146"
Oct 30 18:05:50.446328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.446300    5534 scope.go:122] "RemoveContainer" containerID="c96d8aa4bd95fc3a9d1f5b1cab0b45b940d25ab77cb756ead43517eb7f184dc5"
Oct 30 18:05:50.476575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.476526    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"]
Oct 30 18:05:50.481920 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.481890    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"]
Oct 30 18:05:50.488133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.488107    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-455086/simulator-w1-455086-kserve-db8fcdbfb-xw8j5"]
Oct 30 18:05:50.488286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.488208    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy" containerID="cri-o://8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b" gracePeriod=30
Oct 30 18:05:50.513043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.512967    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:50.524761 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.524729    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"]
Oct 30 18:05:50.529757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.529734    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-w1-455086-kserve-786fcff9d8-nd2g8"]
Oct 30 18:05:50.713943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:50.709824    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:51.140448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.140411    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:51.150734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.150676    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"]
Oct 30 18:05:51.151974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.151931    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f63b2055-6998-4ada-8b98-6986ef43ecc0" containerName="revoke-keys"
Oct 30 18:05:51.151974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.151971    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63b2055-6998-4ada-8b98-6986ef43ecc0" containerName="revoke-keys"
Oct 30 18:05:51.152279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152064    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ff702ec3-0e0d-4cbc-b57e-e5f10437634e" containerName="revoke-keys"
Oct 30 18:05:51.152279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152074    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff702ec3-0e0d-4cbc-b57e-e5f10437634e" containerName="revoke-keys"
Oct 30 18:05:51.152279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152086    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main"
Oct 30 18:05:51.152279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152094    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main"
Oct 30 18:05:51.152279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152109    5534 cpu_manager.go:414] "containerMap: removing container" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main"
Oct 30 18:05:51.152279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152117    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main"
Oct 30 18:05:51.152823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152284    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" containerName="main"
Oct 30 18:05:51.152823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152298    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" containerName="main"
Oct 30 18:05:51.152823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152316    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63b2055-6998-4ada-8b98-6986ef43ecc0" containerName="revoke-keys"
Oct 30 18:05:51.152823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.152337    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff702ec3-0e0d-4cbc-b57e-e5f10437634e" containerName="revoke-keys"
Oct 30 18:05:51.160565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.160537    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"]
Oct 30 18:05:51.160727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.160628    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.220658 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.220603    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.220843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.220758    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.221001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.220975    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cx9\" (UniqueName: \"kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.272275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.272243    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:51.272491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.272472    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:51.321945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.321294    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:51.321945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.321890    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cx9\" (UniqueName: \"kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.322155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.322015    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.322155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.322074    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.324247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.324211    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.360665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.355013    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.373678 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.372474    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cx9\" (UniqueName: \"kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.478340 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.478260    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:51.511607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.511573    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:51.543434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.543410    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"]
Oct 30 18:05:51.548329 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.548302    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-306aed-srm2x"]
Oct 30 18:05:51.855429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.855399    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:51.856260 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.856236    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"]
Oct 30 18:05:51.859857 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:51.859825    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466f653f_8036_49ce_82c7_d27fb7fb9804.slice/crio-bf5766f46887b0c5ebddf76e7201c01d196b7f396007b2173c1341778da96761 WatchSource:0}: Error finding container bf5766f46887b0c5ebddf76e7201c01d196b7f396007b2173c1341778da96761: Status 404 returned error can't find the container with id bf5766f46887b0c5ebddf76e7201c01d196b7f396007b2173c1341778da96761
Oct 30 18:05:51.966769 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.966735    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"]
Oct 30 18:05:51.972288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:51.972255    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:52.090762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.090715    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:52.143375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.143295    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"]
Oct 30 18:05:52.147766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.147729    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-d207f8-l4zjk"]
Oct 30 18:05:52.212046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.212009    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:52.372218 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.372188    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48297df0-703f-4fc9-b82f-fb8a191587e6" path="/var/lib/kubelet/pods/48297df0-703f-4fc9-b82f-fb8a191587e6/volumes"
Oct 30 18:05:52.372626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.372612    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="526a3a13-0745-46f4-a20b-44ee8bceb95b" path="/var/lib/kubelet/pods/526a3a13-0745-46f4-a20b-44ee8bceb95b/volumes"
Oct 30 18:05:52.373013 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.373000    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dceebab-1851-4341-9bb5-1951096b720f" path="/var/lib/kubelet/pods/5dceebab-1851-4341-9bb5-1951096b720f/volumes"
Oct 30 18:05:52.373703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.373689    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53ac57c-cc71-4a25-8d14-97b27b9dedd6" path="/var/lib/kubelet/pods/a53ac57c-cc71-4a25-8d14-97b27b9dedd6/volumes"
Oct 30 18:05:52.432842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.432761    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv" event={"ID":"466f653f-8036-49ce-82c7-d27fb7fb9804","Type":"ContainerStarted","Data":"191a72a4b6ac2fcfe9b78b87bbbefff5673f77e0a194b7e298fdff9e22911fe8"}
Oct 30 18:05:52.432842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.432815    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv" event={"ID":"466f653f-8036-49ce-82c7-d27fb7fb9804","Type":"ContainerStarted","Data":"bf5766f46887b0c5ebddf76e7201c01d196b7f396007b2173c1341778da96761"}
Oct 30 18:05:52.474996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.474938    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv" podStartSLOduration=1.474923205 podStartE2EDuration="1.474923205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:52.473163768 +0000 UTC m=+3158.235207581" watchObservedRunningTime="2026-10-30 18:05:52.474923205 +0000 UTC m=+3158.236967018"
Oct 30 18:05:52.602689 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.602631    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:05:52.647413 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:52.647374    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:53.442408 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:53.442375    5534 generic.go:356] "Generic (PLEG): container finished" podID="466f653f-8036-49ce-82c7-d27fb7fb9804" containerID="191a72a4b6ac2fcfe9b78b87bbbefff5673f77e0a194b7e298fdff9e22911fe8" exitCode=0
Oct 30 18:05:53.442597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:53.442466    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv" event={"ID":"466f653f-8036-49ce-82c7-d27fb7fb9804","Type":"ContainerDied","Data":"191a72a4b6ac2fcfe9b78b87bbbefff5673f77e0a194b7e298fdff9e22911fe8"}
Oct 30 18:05:53.967336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:53.967295    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:54.143862 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.143822    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:54.272185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.271311    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:54.386914 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.386844    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:54.629274 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.629248    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:54.709754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.709712    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca\") pod \"466f653f-8036-49ce-82c7-d27fb7fb9804\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") "
Oct 30 18:05:54.709970 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.709801    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca\") pod \"466f653f-8036-49ce-82c7-d27fb7fb9804\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") "
Oct 30 18:05:54.709970 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.709877    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9\" (UniqueName: \"kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9\") pod \"466f653f-8036-49ce-82c7-d27fb7fb9804\" (UID: \"466f653f-8036-49ce-82c7-d27fb7fb9804\") "
Oct 30 18:05:54.710253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.710220    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca" pod "466f653f-8036-49ce-82c7-d27fb7fb9804" (UID: "466f653f-8036-49ce-82c7-d27fb7fb9804"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:54.710522 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.710466    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:54.712681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.712379    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9" pod "466f653f-8036-49ce-82c7-d27fb7fb9804" (UID: "466f653f-8036-49ce-82c7-d27fb7fb9804"). InnerVolumeSpecName "kube-api-access-57cx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:54.712681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.712481    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca" pod "466f653f-8036-49ce-82c7-d27fb7fb9804" (UID: "466f653f-8036-49ce-82c7-d27fb7fb9804"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:05:54.811996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.811921    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466f653f-8036-49ce-82c7-d27fb7fb9804-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:54.811996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:54.811953    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57cx9\" (UniqueName: \"kubernetes.io/projected/466f653f-8036-49ce-82c7-d27fb7fb9804-kube-api-access-57cx9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.476581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.476544    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv" event={"ID":"466f653f-8036-49ce-82c7-d27fb7fb9804","Type":"ContainerDied","Data":"bf5766f46887b0c5ebddf76e7201c01d196b7f396007b2173c1341778da96761"}
Oct 30 18:05:55.477027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.476605    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5766f46887b0c5ebddf76e7201c01d196b7f396007b2173c1341778da96761"
Oct 30 18:05:55.477027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.476560    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"
Oct 30 18:05:55.786867 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.786838    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 18:05:55.824968 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.824929    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825003    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825030    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825094    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825202    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825251    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:55.825362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825274    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:55.825362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825298    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825343    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825379    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.825677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825441    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:55.825677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825520    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token\") pod \"ac0178d6-695e-4378-a7d3-a046df89a935\" (UID: \"ac0178d6-695e-4378-a7d3-a046df89a935\") "
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.825978    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.826018    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.826440    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.826461    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.826479    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.826494    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.826506    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0178d6-695e-4378-a7d3-a046df89a935-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.828162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.827996    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:05:55.829001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.828977    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "kube-api-access-fn5gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:55.834241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.834214    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:05:55.834241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.834219    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy" pod "ac0178d6-695e-4378-a7d3-a046df89a935" (UID: "ac0178d6-695e-4378-a7d3-a046df89a935"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:05:55.927866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.927835    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.927866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.927863    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5gb\" (UniqueName: \"kubernetes.io/projected/ac0178d6-695e-4378-a7d3-a046df89a935-kube-api-access-fn5gb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.928088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.927883    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0178d6-695e-4378-a7d3-a046df89a935-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:55.928088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:55.927893    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0178d6-695e-4378-a7d3-a046df89a935-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:05:56.063198 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.063101    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:56.487688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.487637    5534 generic.go:356] "Generic (PLEG): container finished" podID="ac0178d6-695e-4378-a7d3-a046df89a935" containerID="8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b" exitCode=0
Oct 30 18:05:56.488180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.487830    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" event={"ID":"ac0178d6-695e-4378-a7d3-a046df89a935","Type":"ContainerDied","Data":"8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b"}
Oct 30 18:05:56.488180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.487877    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7" event={"ID":"ac0178d6-695e-4378-a7d3-a046df89a935","Type":"ContainerDied","Data":"7702fbe4235a89c939204752ab34346d242fa5bf9410179d4ca38b8b7fb9f068"}
Oct 30 18:05:56.488180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.487899    5534 scope.go:122] "RemoveContainer" containerID="8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b"
Oct 30 18:05:56.488569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.488532    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"
Oct 30 18:05:56.501293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.501273    5534 scope.go:122] "RemoveContainer" containerID="8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b"
Oct 30 18:05:56.501613 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:05:56.501591    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b\": container with ID starting with 8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b not found: ID does not exist" containerID="8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b"
Oct 30 18:05:56.501737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.501620    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b"} err="failed to get container status \"8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b\": rpc error: code = NotFound desc = could not find container \"8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b\": container with ID starting with 8430b42aae981757ba3fb75ada3f22e097a4044bc552241d17c785b01d1aca9b not found: ID does not exist"
Oct 30 18:05:56.556274 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.556237    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"]
Oct 30 18:05:56.571501 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.571472    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-nbbd7"]
Oct 30 18:05:56.734120 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734042    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"]
Oct 30 18:05:56.734725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734709    5534 cpu_manager.go:414] "containerMap: removing container" podUID="466f653f-8036-49ce-82c7-d27fb7fb9804" containerName="revoke-keys"
Oct 30 18:05:56.734781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734727    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466f653f-8036-49ce-82c7-d27fb7fb9804" containerName="revoke-keys"
Oct 30 18:05:56.734850 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734781    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy"
Oct 30 18:05:56.734850 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734788    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy"
Oct 30 18:05:56.734936 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734906    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" containerName="istio-proxy"
Oct 30 18:05:56.734936 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.734918    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="466f653f-8036-49ce-82c7-d27fb7fb9804" containerName="revoke-keys"
Oct 30 18:05:56.740165 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.740136    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.755681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.754956    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:05:56.755681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.755569    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:05:56.760066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.759834    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"]
Oct 30 18:05:56.839585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.839527    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvfr\" (UniqueName: \"kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.839910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.839698    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.839910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.839828    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.941345 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.941314    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvfr\" (UniqueName: \"kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.941538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.941398    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.941538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.941505    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.942148 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.942126    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.944355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.944311    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.964977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.964947    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvfr\" (UniqueName: \"kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr\") pod \"maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:56.968048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:56.968022    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerName="payload-processing" containerID="cri-o://040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98" gracePeriod=25
Oct 30 18:05:57.055654 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.055562    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:05:57.364667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.364599    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:05:57.374943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.374917    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:05:57.375082 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.375006    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:05:57.375767 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.375732    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"]
Oct 30 18:05:57.376423 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:05:57.376398    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6395d2_3393_4ad9_9148_b48fe0327add.slice/crio-c5b8e36c3f0a67d573ea0b1c96adb9a74739f1f6b048fe6215a6924ef01d3263 WatchSource:0}: Error finding container c5b8e36c3f0a67d573ea0b1c96adb9a74739f1f6b048fe6215a6924ef01d3263: Status 404 returned error can't find the container with id c5b8e36c3f0a67d573ea0b1c96adb9a74739f1f6b048fe6215a6924ef01d3263
Oct 30 18:05:57.402712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.402682    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:05:57.501493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.501434    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m" event={"ID":"ee6395d2-3393-4ad9-9148-b48fe0327add","Type":"ContainerStarted","Data":"c5b8e36c3f0a67d573ea0b1c96adb9a74739f1f6b048fe6215a6924ef01d3263"}
Oct 30 18:05:57.935077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:57.935039    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:58.371519 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:58.371487    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0178d6-695e-4378-a7d3-a046df89a935" path="/var/lib/kubelet/pods/ac0178d6-695e-4378-a7d3-a046df89a935/volumes"
Oct 30 18:05:58.552798 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:58.552754    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m" event={"ID":"ee6395d2-3393-4ad9-9148-b48fe0327add","Type":"ContainerStarted","Data":"9e73098842bff95a3812431ac82b44d0463054d1c62e21e73d0a276abafb5a12"}
Oct 30 18:05:58.567310 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:58.567280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:58.598882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:58.598827    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m" podStartSLOduration=2.598806625 podStartE2EDuration="2.598806625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:58.593938472 +0000 UTC m=+3164.355982286" watchObservedRunningTime="2026-10-30 18:05:58.598806625 +0000 UTC m=+3164.360850439"
Oct 30 18:05:58.698868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:58.698791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:05:59.561742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:59.561706    5534 generic.go:356] "Generic (PLEG): container finished" podID="ee6395d2-3393-4ad9-9148-b48fe0327add" containerID="9e73098842bff95a3812431ac82b44d0463054d1c62e21e73d0a276abafb5a12" exitCode=0
Oct 30 18:05:59.562167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:59.561798    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m" event={"ID":"ee6395d2-3393-4ad9-9148-b48fe0327add","Type":"ContainerDied","Data":"9e73098842bff95a3812431ac82b44d0463054d1c62e21e73d0a276abafb5a12"}
Oct 30 18:05:59.899286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:05:59.899252    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:00.027397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.027360    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:00.236412 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.236318    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"]
Oct 30 18:06:00.236625 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.236600    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy" containerID="cri-o://1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97" gracePeriod=30
Oct 30 18:06:00.295384 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.295352    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:00.434399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.434368    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:00.728861 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.728832    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:06:00.760786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.760751    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:00.800012 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.799978    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca\") pod \"ee6395d2-3393-4ad9-9148-b48fe0327add\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") "
Oct 30 18:06:00.800219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.800073    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca\") pod \"ee6395d2-3393-4ad9-9148-b48fe0327add\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") "
Oct 30 18:06:00.800219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.800134    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr\" (UniqueName: \"kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr\") pod \"ee6395d2-3393-4ad9-9148-b48fe0327add\" (UID: \"ee6395d2-3393-4ad9-9148-b48fe0327add\") "
Oct 30 18:06:00.800442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.800412    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca" pod "ee6395d2-3393-4ad9-9148-b48fe0327add" (UID: "ee6395d2-3393-4ad9-9148-b48fe0327add"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:00.800754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.800730    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:00.805118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.803198    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr" pod "ee6395d2-3393-4ad9-9148-b48fe0327add" (UID: "ee6395d2-3393-4ad9-9148-b48fe0327add"). InnerVolumeSpecName "kube-api-access-ngvfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:00.805118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.803464    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca" pod "ee6395d2-3393-4ad9-9148-b48fe0327add" (UID: "ee6395d2-3393-4ad9-9148-b48fe0327add"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:00.902432 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.902392    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6395d2-3393-4ad9-9148-b48fe0327add-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:00.902432 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.902426    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvfr\" (UniqueName: \"kubernetes.io/projected/ee6395d2-3393-4ad9-9148-b48fe0327add-kube-api-access-ngvfr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:00.915585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.915552    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"]
Oct 30 18:06:00.916227 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.916204    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ee6395d2-3393-4ad9-9148-b48fe0327add" containerName="revoke-keys"
Oct 30 18:06:00.916227 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.916222    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6395d2-3393-4ad9-9148-b48fe0327add" containerName="revoke-keys"
Oct 30 18:06:00.916396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.916349    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6395d2-3393-4ad9-9148-b48fe0327add" containerName="revoke-keys"
Oct 30 18:06:00.920022 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.920005    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:00.939440 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.939412    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"]
Oct 30 18:06:00.990517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.990483    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"]
Oct 30 18:06:00.990870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:00.990733    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy" containerID="cri-o://3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a" gracePeriod=30
Oct 30 18:06:01.003272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.003240    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.003490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.003464    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8629r\" (UniqueName: \"kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.003603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.003585    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.104360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.104326    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8629r\" (UniqueName: \"kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.104562 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.104406    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.104562 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.104527    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.105298 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.105267    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.106974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.106951    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.127615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.127588    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8629r\" (UniqueName: \"kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r\") pod \"maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.234890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.234810    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:01.548880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.548795    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"]
Oct 30 18:06:01.554160 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:06:01.554129    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffdd6ef_dbf0_4515_b509_c216c7de4882.slice/crio-1deb12af52745137d34c9e419987537c2885ddc87d5f4f6456b810c49cc6e401 WatchSource:0}: Error finding container 1deb12af52745137d34c9e419987537c2885ddc87d5f4f6456b810c49cc6e401: Status 404 returned error can't find the container with id 1deb12af52745137d34c9e419987537c2885ddc87d5f4f6456b810c49cc6e401
Oct 30 18:06:01.586245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.586212    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb" event={"ID":"1ffdd6ef-dbf0-4515-b509-c216c7de4882","Type":"ContainerStarted","Data":"1deb12af52745137d34c9e419987537c2885ddc87d5f4f6456b810c49cc6e401"}
Oct 30 18:06:01.587771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.587749    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"
Oct 30 18:06:01.587921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.587777    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m" event={"ID":"ee6395d2-3393-4ad9-9148-b48fe0327add","Type":"ContainerDied","Data":"c5b8e36c3f0a67d573ea0b1c96adb9a74739f1f6b048fe6215a6924ef01d3263"}
Oct 30 18:06:01.587921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:01.587819    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5b8e36c3f0a67d573ea0b1c96adb9a74739f1f6b048fe6215a6924ef01d3263"
Oct 30 18:06:02.594122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:02.594091    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:06:02.600149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:02.600092    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb" event={"ID":"1ffdd6ef-dbf0-4515-b509-c216c7de4882","Type":"ContainerStarted","Data":"211cc7dd7c85caa1348996cfe547c835e38ae4df5bc8e9d8eda3da97f931cb06"}
Oct 30 18:06:02.671977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:02.671925    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb" podStartSLOduration=2.67190991 podStartE2EDuration="2.67190991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:02.668809569 +0000 UTC m=+3168.430853383" watchObservedRunningTime="2026-10-30 18:06:02.67190991 +0000 UTC m=+3168.433953722"
Oct 30 18:06:02.721691 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:02.721627    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:02.841011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:02.840906    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:03.609816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:03.609775    5534 generic.go:356] "Generic (PLEG): container finished" podID="1ffdd6ef-dbf0-4515-b509-c216c7de4882" containerID="211cc7dd7c85caa1348996cfe547c835e38ae4df5bc8e9d8eda3da97f931cb06" exitCode=0
Oct 30 18:06:03.610256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:03.609848    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb" event={"ID":"1ffdd6ef-dbf0-4515-b509-c216c7de4882","Type":"ContainerDied","Data":"211cc7dd7c85caa1348996cfe547c835e38ae4df5bc8e9d8eda3da97f931cb06"}
Oct 30 18:06:04.141790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.141756    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:04.712931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.712898    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:04.791191 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.791161    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:04.838675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.838608    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:04.869272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.869228    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca\") pod \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") "
Oct 30 18:06:04.869483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.869349    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r\" (UniqueName: \"kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r\") pod \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") "
Oct 30 18:06:04.869483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.869396    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca\") pod \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\" (UID: \"1ffdd6ef-dbf0-4515-b509-c216c7de4882\") "
Oct 30 18:06:04.870022 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.869943    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca" pod "1ffdd6ef-dbf0-4515-b509-c216c7de4882" (UID: "1ffdd6ef-dbf0-4515-b509-c216c7de4882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:04.870507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.870473    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:04.872590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.872561    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca" pod "1ffdd6ef-dbf0-4515-b509-c216c7de4882" (UID: "1ffdd6ef-dbf0-4515-b509-c216c7de4882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:04.873206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.873180    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r" pod "1ffdd6ef-dbf0-4515-b509-c216c7de4882" (UID: "1ffdd6ef-dbf0-4515-b509-c216c7de4882"). InnerVolumeSpecName "kube-api-access-8629r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:04.898582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.898542    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"]
Oct 30 18:06:04.903317 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.903281    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8d1062-r2xht"]
Oct 30 18:06:04.961138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.960787    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:04.972246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.972212    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8629r\" (UniqueName: \"kubernetes.io/projected/1ffdd6ef-dbf0-4515-b509-c216c7de4882-kube-api-access-8629r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:04.972246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:04.972244    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd6ef-dbf0-4515-b509-c216c7de4882-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.158135 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.158106    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 18:06:05.276805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.276767    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume\") pod \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") "
Oct 30 18:06:05.277018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.276826    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk\" (UniqueName: \"kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk\") pod \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\" (UID: \"2f8f8383-1e7b-4d07-9993-3fe0e4385b02\") "
Oct 30 18:06:05.277554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.277511    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume" pod "2f8f8383-1e7b-4d07-9993-3fe0e4385b02" (UID: "2f8f8383-1e7b-4d07-9993-3fe0e4385b02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:05.280355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.280324    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk" pod "2f8f8383-1e7b-4d07-9993-3fe0e4385b02" (UID: "2f8f8383-1e7b-4d07-9993-3fe0e4385b02"). InnerVolumeSpecName "kube-api-access-zsphk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:05.351821 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.351786    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:05.378543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.378509    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.378543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.378539    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsphk\" (UniqueName: \"kubernetes.io/projected/2f8f8383-1e7b-4d07-9993-3fe0e4385b02-kube-api-access-zsphk\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.506586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.506555    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 18:06:05.581780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.581743    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.581793    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.581844    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.581883    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582032    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582105    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:05.582234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582115    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582203    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582280    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582386    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2\") pod \"88e416bc-4744-4354-b782-9ae4e6c64217\" (UID: \"88e416bc-4744-4354-b782-9ae4e6c64217\") "
Oct 30 18:06:05.582745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.582694    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:05.583273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583003    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:05.583273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583189    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:05.583532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583459    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:05.583955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583928    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.584074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583960    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.584074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583975    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.584074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.583989    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e416bc-4744-4354-b782-9ae4e6c64217-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.584074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.584002    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.587176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.587142    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:06:05.587176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.587157    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "kube-api-access-prdd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:05.587317 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.587220    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:05.588823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.588798    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token" pod "88e416bc-4744-4354-b782-9ae4e6c64217" (UID: "88e416bc-4744-4354-b782-9ae4e6c64217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:05.631283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.631250    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"
Oct 30 18:06:05.631984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.631255    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb" event={"ID":"1ffdd6ef-dbf0-4515-b509-c216c7de4882","Type":"ContainerDied","Data":"1deb12af52745137d34c9e419987537c2885ddc87d5f4f6456b810c49cc6e401"}
Oct 30 18:06:05.632079 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.632009    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1deb12af52745137d34c9e419987537c2885ddc87d5f4f6456b810c49cc6e401"
Oct 30 18:06:05.633280 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.633252    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerID="3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84" exitCode=0
Oct 30 18:06:05.633421 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.633346    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"
Oct 30 18:06:05.633421 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.633374    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" event={"ID":"2f8f8383-1e7b-4d07-9993-3fe0e4385b02","Type":"ContainerDied","Data":"3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84"}
Oct 30 18:06:05.633421 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.633414    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp" event={"ID":"2f8f8383-1e7b-4d07-9993-3fe0e4385b02","Type":"ContainerDied","Data":"2bea7932978bd641a544018543a2488fd9c55fead863941c2073d4698fd41eff"}
Oct 30 18:06:05.633577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.633443    5534 scope.go:122] "RemoveContainer" containerID="3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84"
Oct 30 18:06:05.635548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.635527    5534 generic.go:356] "Generic (PLEG): container finished" podID="88e416bc-4744-4354-b782-9ae4e6c64217" containerID="1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97" exitCode=0
Oct 30 18:06:05.635708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.635590    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"
Oct 30 18:06:05.635708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.635667    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" event={"ID":"88e416bc-4744-4354-b782-9ae4e6c64217","Type":"ContainerDied","Data":"1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97"}
Oct 30 18:06:05.635808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.635722    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg" event={"ID":"88e416bc-4744-4354-b782-9ae4e6c64217","Type":"ContainerDied","Data":"e9d22f12cb4a73c163fddc4f268c709c7ea379a0843b768a1f5e226393e6fabd"}
Oct 30 18:06:05.646376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.646355    5534 scope.go:122] "RemoveContainer" containerID="3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84"
Oct 30 18:06:05.646802 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:05.646777    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84\": container with ID starting with 3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84 not found: ID does not exist" containerID="3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84"
Oct 30 18:06:05.646889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.646811    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84"} err="failed to get container status \"3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84\": rpc error: code = NotFound desc = could not find container \"3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84\": container with ID starting with 3e6b932b9884db7b920d66a9e12ae8964108c0c2bcd8f5e258d145c3dc683f84 not found: ID does not exist"
Oct 30 18:06:05.646889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.646830    5534 scope.go:122] "RemoveContainer" containerID="1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97"
Oct 30 18:06:05.659537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.659515    5534 scope.go:122] "RemoveContainer" containerID="2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"
Oct 30 18:06:05.672700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.672680    5534 scope.go:122] "RemoveContainer" containerID="1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97"
Oct 30 18:06:05.673018 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:05.672995    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97\": container with ID starting with 1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97 not found: ID does not exist" containerID="1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97"
Oct 30 18:06:05.673096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.673027    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97"} err="failed to get container status \"1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97\": rpc error: code = NotFound desc = could not find container \"1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97\": container with ID starting with 1905452989f31010a6b25f773c0e3f0bc24380251eaf16dd98034f7830fc2e97 not found: ID does not exist"
Oct 30 18:06:05.673096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.673045    5534 scope.go:122] "RemoveContainer" containerID="2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"
Oct 30 18:06:05.673334 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:05.673316    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a\": container with ID starting with 2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a not found: ID does not exist" containerID="2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"
Oct 30 18:06:05.673399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.673341    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a"} err="failed to get container status \"2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a\": rpc error: code = NotFound desc = could not find container \"2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a\": container with ID starting with 2dc916bcbc7d3cf2677878ab5b2206b59b535b08cc3ad55ac3a7f67c7af6e66a not found: ID does not exist"
Oct 30 18:06:05.685400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.685374    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e416bc-4744-4354-b782-9ae4e6c64217-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.685400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.685398    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.685400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.685407    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e416bc-4744-4354-b782-9ae4e6c64217-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.685713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.685416    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdd2\" (UniqueName: \"kubernetes.io/projected/88e416bc-4744-4354-b782-9ae4e6c64217-kube-api-access-prdd2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:05.700481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.700449    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"]
Oct 30 18:06:05.704654 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.704617    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-455086-5f7bc59956-g52bp"]
Oct 30 18:06:05.718780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.718753    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"]
Oct 30 18:06:05.736084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.736041    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"]
Oct 30 18:06:05.745941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.745898    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"]
Oct 30 18:06:05.763167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.763132    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"]
Oct 30 18:06:05.768878 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.768844    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-2mkqg"]
Oct 30 18:06:05.853786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:05.853751    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:06.279570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.279539    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 18:06:06.372563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.372525    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" path="/var/lib/kubelet/pods/2f8f8383-1e7b-4d07-9993-3fe0e4385b02/volumes"
Oct 30 18:06:06.373318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.373288    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" path="/var/lib/kubelet/pods/88e416bc-4744-4354-b782-9ae4e6c64217/volumes"
Oct 30 18:06:06.374666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.373965    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e715a0-78dc-4d01-b5bd-f7116e791ee6" path="/var/lib/kubelet/pods/f6e715a0-78dc-4d01-b5bd-f7116e791ee6/volumes"
Oct 30 18:06:06.393569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393527    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.393790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393705    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.393790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393740    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.393790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393770    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.393994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393843    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.393994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393905    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.393994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.393937    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.394162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.394030    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.394162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.394067    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token\") pod \"0c6a8730-f4fa-4440-8994-5cad740314f1\" (UID: \"0c6a8730-f4fa-4440-8994-5cad740314f1\") "
Oct 30 18:06:06.394310 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.394279    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:06.394796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.394768    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.395172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.395147    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:06.395920 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.395868    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:06.396230 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.396200    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:06.396337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.396259    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:06.397586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.397559    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:06.397719 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.397626    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:06.398391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.398366    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "kube-api-access-bmqvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:06.398465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.398405    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo" pod "0c6a8730-f4fa-4440-8994-5cad740314f1" (UID: "0c6a8730-f4fa-4440-8994-5cad740314f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496150    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496183    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqvz\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-kube-api-access-bmqvz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496198    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496210    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496224    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496235    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496247    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6a8730-f4fa-4440-8994-5cad740314f1-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.496531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.496259    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6a8730-f4fa-4440-8994-5cad740314f1-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:06.647241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.647204    5534 generic.go:356] "Generic (PLEG): container finished" podID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerID="3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a" exitCode=0
Oct 30 18:06:06.647241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.647247    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"
Oct 30 18:06:06.647241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.647257    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" event={"ID":"0c6a8730-f4fa-4440-8994-5cad740314f1","Type":"ContainerDied","Data":"3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a"}
Oct 30 18:06:06.647609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.647306    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt" event={"ID":"0c6a8730-f4fa-4440-8994-5cad740314f1","Type":"ContainerDied","Data":"33dee7a610d7068aec808c85ab2ef4a9035bc85d0241b5dc4b045564eb982f3b"}
Oct 30 18:06:06.647609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.647334    5534 scope.go:122] "RemoveContainer" containerID="3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a"
Oct 30 18:06:06.662740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.662713    5534 scope.go:122] "RemoveContainer" containerID="3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a"
Oct 30 18:06:06.663533 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:06.663507    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a\": container with ID starting with 3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a not found: ID does not exist" containerID="3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a"
Oct 30 18:06:06.663672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.663545    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a"} err="failed to get container status \"3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a\": rpc error: code = NotFound desc = could not find container \"3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a\": container with ID starting with 3d06f2e2c886af28aef115498060321c538c1cb63a02504d034d5a8a34763c5a not found: ID does not exist"
Oct 30 18:06:06.722353 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.722320    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"]
Oct 30 18:06:06.733753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.733716    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-4lsjt"]
Oct 30 18:06:06.977567 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:06.977469    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:07.364472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:07.364427    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:06:07.403124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:07.403087    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:06:08.370236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:08.370200    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" path="/var/lib/kubelet/pods/0c6a8730-f4fa-4440-8994-5cad740314f1/volumes"
Oct 30 18:06:08.705896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:08.705814    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:08.957851 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:08.954853    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:09.111493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.111471    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 18:06:09.229352 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.229256    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6\" (UniqueName: \"kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6\") pod \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") "
Oct 30 18:06:09.229352 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.229340    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume\") pod \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\" (UID: \"7f35eae8-b2d2-4c94-98e4-de92f0efa592\") "
Oct 30 18:06:09.229744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.229716    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume" pod "7f35eae8-b2d2-4c94-98e4-de92f0efa592" (UID: "7f35eae8-b2d2-4c94-98e4-de92f0efa592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:09.231593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.231567    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6" pod "7f35eae8-b2d2-4c94-98e4-de92f0efa592" (UID: "7f35eae8-b2d2-4c94-98e4-de92f0efa592"). InnerVolumeSpecName "kube-api-access-pxjr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:09.330915 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.330883    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxjr6\" (UniqueName: \"kubernetes.io/projected/7f35eae8-b2d2-4c94-98e4-de92f0efa592-kube-api-access-pxjr6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:09.330915 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.330913    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35eae8-b2d2-4c94-98e4-de92f0efa592-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:09.689546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.689505    5534 generic.go:356] "Generic (PLEG): container finished" podID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerID="b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d" exitCode=0
Oct 30 18:06:09.689546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.689542    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"
Oct 30 18:06:09.690090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.689579    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" event={"ID":"7f35eae8-b2d2-4c94-98e4-de92f0efa592","Type":"ContainerDied","Data":"b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d"}
Oct 30 18:06:09.690090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.689669    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x" event={"ID":"7f35eae8-b2d2-4c94-98e4-de92f0efa592","Type":"ContainerDied","Data":"4123b4ca71b22119f198bc8f8a1e52bcade44360383237c97f118a028950f6d0"}
Oct 30 18:06:09.690090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.689673    5534 scope.go:122] "RemoveContainer" containerID="b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d"
Oct 30 18:06:09.702089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.702066    5534 scope.go:122] "RemoveContainer" containerID="b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d"
Oct 30 18:06:09.702419 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:09.702400    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d\": container with ID starting with b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d not found: ID does not exist" containerID="b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d"
Oct 30 18:06:09.702476 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.702431    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d"} err="failed to get container status \"b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d\": rpc error: code = NotFound desc = could not find container \"b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d\": container with ID starting with b6d10c147138924ba3ef4e9ac7df60fab9faf57d29989b187336bd0c5675264d not found: ID does not exist"
Oct 30 18:06:09.742103 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.742063    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"]
Oct 30 18:06:09.754363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:09.754318    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-d207f8-669f75f5f-xz24x"]
Oct 30 18:06:10.373281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:10.373242    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" path="/var/lib/kubelet/pods/7f35eae8-b2d2-4c94-98e4-de92f0efa592/volumes"
Oct 30 18:06:10.470753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:10.470715    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:11.458016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:11.457970    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"]
Oct 30 18:06:11.568905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:11.568843    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:11.683600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:11.683565    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:11.800609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:11.800505    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:11.913542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:11.913508    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:12.430496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:12.430464    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:12.808873 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:12.808785    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:13.554238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:13.554202    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:13.754818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:13.754784    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:13.922470 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:13.922419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:14.423258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423211    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"]
Oct 30 18:06:14.423840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423823    5534 cpu_manager.go:414] "containerMap: removing container" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy"
Oct 30 18:06:14.423840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423839    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423853    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerName="payload-pre-processing"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423858    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerName="payload-pre-processing"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423870    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerName="payload-processing"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423875    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerName="payload-processing"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423894    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1ffdd6ef-dbf0-4515-b509-c216c7de4882" containerName="revoke-keys"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423898    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffdd6ef-dbf0-4515-b509-c216c7de4882" containerName="revoke-keys"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423917    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.423922    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.424000    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffdd6ef-dbf0-4515-b509-c216c7de4882" containerName="revoke-keys"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.424010    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.424018    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a8730-f4fa-4440-8994-5cad740314f1" containerName="istio-proxy"
Oct 30 18:06:14.424021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.424028    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f35eae8-b2d2-4c94-98e4-de92f0efa592" containerName="payload-pre-processing"
Oct 30 18:06:14.424707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.424035    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8f8383-1e7b-4d07-9993-3fe0e4385b02" containerName="payload-processing"
Oct 30 18:06:14.427552 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.427528    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.450321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.450287    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"]
Oct 30 18:06:14.521257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.521227    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:06:14.521435 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.521327    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:06:14.619839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.619799    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.620047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.619942    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spk5r\" (UniqueName: \"kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.620047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.620038    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.721333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.721234    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.721333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.721317    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spk5r\" (UniqueName: \"kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.721593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.721375    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.722307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.722263    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.723984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.723952    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.744945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.744908    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spk5r\" (UniqueName: \"kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:14.843304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.843265    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"]
Oct 30 18:06:14.843962 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.843947    5534 cpu_manager.go:414] "containerMap: removing container" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy"
Oct 30 18:06:14.844035 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.843964    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy"
Oct 30 18:06:14.844093 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.844084    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e416bc-4744-4354-b782-9ae4e6c64217" containerName="istio-proxy"
Oct 30 18:06:14.847977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.847956    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:14.864999 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.864894    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d323f5-serving-cert\""
Oct 30 18:06:14.870233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.870204    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"]
Oct 30 18:06:14.969314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:14.969280    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:15.029819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.029713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.029819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.029776    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.030060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.029865    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.030060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.029985    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gjl\" (UniqueName: \"kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.042491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.042464    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:15.091411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.088898    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:15.133978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.133948    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.134087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.134031    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.134087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.134060    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.134189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.134127    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gjl\" (UniqueName: \"kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.134904 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.134880    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.137461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.137435    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.137637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.137569    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.176122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.176086    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gjl\" (UniqueName: \"kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl\") pod \"maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.246295 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:15.246257    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:06:15.380054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.380018    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"]
Oct 30 18:06:15.384027 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:06:15.383998    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679ed575_94a4_4801_a458_41f6c66d1631.slice/crio-0346ab06683d2ff11ed0def6c6377771066ead5f392ccafa0d4c54156eb954f0 WatchSource:0}: Error finding container 0346ab06683d2ff11ed0def6c6377771066ead5f392ccafa0d4c54156eb954f0: Status 404 returned error can't find the container with id 0346ab06683d2ff11ed0def6c6377771066ead5f392ccafa0d4c54156eb954f0
Oct 30 18:06:15.466227 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.466195    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:15.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.777285    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv" event={"ID":"679ed575-94a4-4801-a458-41f6c66d1631","Type":"ContainerStarted","Data":"d44036016adbb01bab421b7d57a9a372f2bdafcd6741394ebfc4b0d928b463ae"}
Oct 30 18:06:15.784876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.777364    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv" event={"ID":"679ed575-94a4-4801-a458-41f6c66d1631","Type":"ContainerStarted","Data":"0346ab06683d2ff11ed0def6c6377771066ead5f392ccafa0d4c54156eb954f0"}
Oct 30 18:06:15.817846 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.817805    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"]
Oct 30 18:06:15.819006 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:06:15.818852    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b7764c_3fd0_476f_babd_3ec5b7ba335d.slice/crio-8b9298998d31cd2e239f1a1406d0004cbc790cfee8c7de9b599daa4f00d2c8f6 WatchSource:0}: Error finding container 8b9298998d31cd2e239f1a1406d0004cbc790cfee8c7de9b599daa4f00d2c8f6: Status 404 returned error can't find the container with id 8b9298998d31cd2e239f1a1406d0004cbc790cfee8c7de9b599daa4f00d2c8f6
Oct 30 18:06:15.821161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.820820    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:06:15.827614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:15.827573    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv" podStartSLOduration=1.8275608509999999 podStartE2EDuration="1.827560851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:15.819087293 +0000 UTC m=+3181.581131107" watchObservedRunningTime="2026-10-30 18:06:15.827560851 +0000 UTC m=+3181.589604664"
Oct 30 18:06:16.458675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.458549    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" podUID="841b5998-a749-4267-9df7-64ff045cc236" containerName="payload-pre-processing" containerID="cri-o://c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15" gracePeriod=25
Oct 30 18:06:16.793570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.793474    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf" event={"ID":"f3b7764c-3fd0-476f-babd-3ec5b7ba335d","Type":"ContainerStarted","Data":"7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca"}
Oct 30 18:06:16.793570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.793541    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf" event={"ID":"f3b7764c-3fd0-476f-babd-3ec5b7ba335d","Type":"ContainerStarted","Data":"8b9298998d31cd2e239f1a1406d0004cbc790cfee8c7de9b599daa4f00d2c8f6"}
Oct 30 18:06:16.794537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.794468    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:16.795785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.795755    5534 generic.go:356] "Generic (PLEG): container finished" podID="679ed575-94a4-4801-a458-41f6c66d1631" containerID="d44036016adbb01bab421b7d57a9a372f2bdafcd6741394ebfc4b0d928b463ae" exitCode=0
Oct 30 18:06:16.795887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.795803    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv" event={"ID":"679ed575-94a4-4801-a458-41f6c66d1631","Type":"ContainerDied","Data":"d44036016adbb01bab421b7d57a9a372f2bdafcd6741394ebfc4b0d928b463ae"}
Oct 30 18:06:16.836897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:16.836842    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf" podStartSLOduration=2.502194982 podStartE2EDuration="2.83682543s" totalImagesPullingTime="334.630448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:16.83518632 +0000 UTC m=+3182.597230134" watchObservedRunningTime="2026-10-30 18:06:16.83682543 +0000 UTC m=+3182.598869242"
Oct 30 18:06:17.261264 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.261228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:17.328690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.328607    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 18:06:17.371775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.371742    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb\" (UniqueName: \"kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb\") pod \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") "
Oct 30 18:06:17.371948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.371899    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume\") pod \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\" (UID: \"cc2a645e-dbe2-4182-beff-2b3e9d59bab3\") "
Oct 30 18:06:17.372687 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.372660    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume" pod "cc2a645e-dbe2-4182-beff-2b3e9d59bab3" (UID: "cc2a645e-dbe2-4182-beff-2b3e9d59bab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:17.375302 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.375274    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb" pod "cc2a645e-dbe2-4182-beff-2b3e9d59bab3" (UID: "cc2a645e-dbe2-4182-beff-2b3e9d59bab3"). InnerVolumeSpecName "kube-api-access-zddfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:17.473662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.473610    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddfb\" (UniqueName: \"kubernetes.io/projected/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-kube-api-access-zddfb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:17.473662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.473668    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2a645e-dbe2-4182-beff-2b3e9d59bab3-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:17.804844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.804754    5534 generic.go:356] "Generic (PLEG): container finished" podID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerID="040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98" exitCode=0
Oct 30 18:06:17.804844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.804819    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"
Oct 30 18:06:17.805089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.804842    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" event={"ID":"cc2a645e-dbe2-4182-beff-2b3e9d59bab3","Type":"ContainerDied","Data":"040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98"}
Oct 30 18:06:17.805089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.804897    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4" event={"ID":"cc2a645e-dbe2-4182-beff-2b3e9d59bab3","Type":"ContainerDied","Data":"6561e121d6f7b3a086a9694bdd2a519cc3c67275172d1d6821bb714519f4379d"}
Oct 30 18:06:17.805089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.804919    5534 scope.go:122] "RemoveContainer" containerID="040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98"
Oct 30 18:06:17.818177 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.818155    5534 scope.go:122] "RemoveContainer" containerID="040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98"
Oct 30 18:06:17.818472 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:17.818452    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98\": container with ID starting with 040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98 not found: ID does not exist" containerID="040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98"
Oct 30 18:06:17.818538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.818480    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98"} err="failed to get container status \"040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98\": rpc error: code = NotFound desc = could not find container \"040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98\": container with ID starting with 040820a3a3cab1271f2b15ca3f8310182574e66e570fee217b3c990db40aff98 not found: ID does not exist"
Oct 30 18:06:17.861674 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.861569    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"]
Oct 30 18:06:17.871351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.871321    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8d1062-785f75c57d-4fxl4"]
Oct 30 18:06:17.968907 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.968876    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:17.979977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.979943    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r\" (UniqueName: \"kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r\") pod \"679ed575-94a4-4801-a458-41f6c66d1631\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") "
Oct 30 18:06:17.980156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.980011    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca\") pod \"679ed575-94a4-4801-a458-41f6c66d1631\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") "
Oct 30 18:06:17.980156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.980057    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca\") pod \"679ed575-94a4-4801-a458-41f6c66d1631\" (UID: \"679ed575-94a4-4801-a458-41f6c66d1631\") "
Oct 30 18:06:17.980469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.980443    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca" pod "679ed575-94a4-4801-a458-41f6c66d1631" (UID: "679ed575-94a4-4801-a458-41f6c66d1631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:17.980944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.980922    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:17.982324 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.982299    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca" pod "679ed575-94a4-4801-a458-41f6c66d1631" (UID: "679ed575-94a4-4801-a458-41f6c66d1631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:17.982689 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:17.982668    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r" pod "679ed575-94a4-4801-a458-41f6c66d1631" (UID: "679ed575-94a4-4801-a458-41f6c66d1631"). InnerVolumeSpecName "kube-api-access-spk5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:18.082323 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.082229    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spk5r\" (UniqueName: \"kubernetes.io/projected/679ed575-94a4-4801-a458-41f6c66d1631-kube-api-access-spk5r\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:18.082323 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.082260    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679ed575-94a4-4801-a458-41f6c66d1631-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:18.372331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.372294    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" path="/var/lib/kubelet/pods/cc2a645e-dbe2-4182-beff-2b3e9d59bab3/volumes"
Oct 30 18:06:18.745871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.745761    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:18.818327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.818277    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv" event={"ID":"679ed575-94a4-4801-a458-41f6c66d1631","Type":"ContainerDied","Data":"0346ab06683d2ff11ed0def6c6377771066ead5f392ccafa0d4c54156eb954f0"}
Oct 30 18:06:18.818543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.818388    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0346ab06683d2ff11ed0def6c6377771066ead5f392ccafa0d4c54156eb954f0"
Oct 30 18:06:18.818543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.818480    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"
Oct 30 18:06:18.862221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:18.862175    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:19.300094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:19.300053    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:20.412850 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.412812    5534 scope.go:122] "RemoveContainer" containerID="d5e5ed6ca68db28c7d25119373641b2ab797bd227da5525c4f06e9bfc2f967e1"
Oct 30 18:06:20.424534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.424515    5534 scope.go:122] "RemoveContainer" containerID="7b6d8c4f2806804df14123bddbfbe7bb29172de7fbb9efcef0e275ad9b827327"
Oct 30 18:06:20.434745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.434722    5534 scope.go:122] "RemoveContainer" containerID="66ec69abad22e2143cc6df2b80434216223246f069e53008f65c166e39a93407"
Oct 30 18:06:20.445564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.445543    5534 scope.go:122] "RemoveContainer" containerID="ce825ac94316fef0ea77e9e8280885148b55466a84ed2267d10618cbd6d56df0"
Oct 30 18:06:20.741265 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.741189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:20.743733 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.743677    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main" containerID="cri-o://59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7" gracePeriod=45
Oct 30 18:06:20.762573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.762508    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main" containerID="cri-o://6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83" gracePeriod=45
Oct 30 18:06:20.773460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:20.773415    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main" containerID="cri-o://e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2" gracePeriod=45
Oct 30 18:06:21.135695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.135253    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 18:06:21.192329 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.192295    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 18:06:21.199432 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.199395    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 18:06:21.224080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224044    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74\" (UniqueName: \"kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74\") pod \"87554784-d0fe-4228-8dcf-c9a0af78a20f\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") "
Oct 30 18:06:21.224289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224097    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir\") pod \"87554784-d0fe-4228-8dcf-c9a0af78a20f\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") "
Oct 30 18:06:21.224289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224140    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs\") pod \"8b804fac-5e8d-4b4f-a194-d17214118b8b\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") "
Oct 30 18:06:21.224289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224217    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir\") pod \"bcdefec2-1662-4bc5-b11b-308df189a01c\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") "
Oct 30 18:06:21.224289 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224270    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm\") pod \"87554784-d0fe-4228-8dcf-c9a0af78a20f\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") "
Oct 30 18:06:21.224529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224302    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache\") pod \"8b804fac-5e8d-4b4f-a194-d17214118b8b\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") "
Oct 30 18:06:21.224529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224322    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home\") pod \"8b804fac-5e8d-4b4f-a194-d17214118b8b\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") "
Oct 30 18:06:21.224529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224342    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home\") pod \"bcdefec2-1662-4bc5-b11b-308df189a01c\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") "
Oct 30 18:06:21.224529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224381    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv\" (UniqueName: \"kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv\") pod \"8b804fac-5e8d-4b4f-a194-d17214118b8b\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") "
Oct 30 18:06:21.224529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224421    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home\") pod \"87554784-d0fe-4228-8dcf-c9a0af78a20f\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") "
Oct 30 18:06:21.224854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224744    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home" pod "8b804fac-5e8d-4b4f-a194-d17214118b8b" (UID: "8b804fac-5e8d-4b4f-a194-d17214118b8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.224854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224752    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home" pod "bcdefec2-1662-4bc5-b11b-308df189a01c" (UID: "bcdefec2-1662-4bc5-b11b-308df189a01c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.224854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224763    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache" pod "8b804fac-5e8d-4b4f-a194-d17214118b8b" (UID: "8b804fac-5e8d-4b4f-a194-d17214118b8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.224854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224821    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir" pod "87554784-d0fe-4228-8dcf-c9a0af78a20f" (UID: "87554784-d0fe-4228-8dcf-c9a0af78a20f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.224854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224850    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache\") pod \"bcdefec2-1662-4bc5-b11b-308df189a01c\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224926    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs\") pod \"bcdefec2-1662-4bc5-b11b-308df189a01c\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224977    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm\") pod \"8b804fac-5e8d-4b4f-a194-d17214118b8b\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225010    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j\" (UniqueName: \"kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j\") pod \"bcdefec2-1662-4bc5-b11b-308df189a01c\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225105    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm\") pod \"bcdefec2-1662-4bc5-b11b-308df189a01c\" (UID: \"bcdefec2-1662-4bc5-b11b-308df189a01c\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.224977    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir" pod "bcdefec2-1662-4bc5-b11b-308df189a01c" (UID: "bcdefec2-1662-4bc5-b11b-308df189a01c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225075    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home" pod "87554784-d0fe-4228-8dcf-c9a0af78a20f" (UID: "87554784-d0fe-4228-8dcf-c9a0af78a20f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225102    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache" pod "bcdefec2-1662-4bc5-b11b-308df189a01c" (UID: "bcdefec2-1662-4bc5-b11b-308df189a01c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225138    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir\") pod \"8b804fac-5e8d-4b4f-a194-d17214118b8b\" (UID: \"8b804fac-5e8d-4b4f-a194-d17214118b8b\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225199    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs\" (UniqueName: \"kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs\") pod \"87554784-d0fe-4228-8dcf-c9a0af78a20f\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") "
Oct 30 18:06:21.225305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225241    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache\") pod \"87554784-d0fe-4228-8dcf-c9a0af78a20f\" (UID: \"87554784-d0fe-4228-8dcf-c9a0af78a20f\") "
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225324    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir" pod "8b804fac-5e8d-4b4f-a194-d17214118b8b" (UID: "8b804fac-5e8d-4b4f-a194-d17214118b8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225828    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225851    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225867    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225878    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225891    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225903    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225915    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.225930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.225927    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.226462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.226124    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache" pod "87554784-d0fe-4228-8dcf-c9a0af78a20f" (UID: "87554784-d0fe-4228-8dcf-c9a0af78a20f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.227781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.227747    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j" pod "bcdefec2-1662-4bc5-b11b-308df189a01c" (UID: "bcdefec2-1662-4bc5-b11b-308df189a01c"). InnerVolumeSpecName "kube-api-access-nm76j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:21.227942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.227784    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm" pod "8b804fac-5e8d-4b4f-a194-d17214118b8b" (UID: "8b804fac-5e8d-4b4f-a194-d17214118b8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.228453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.228414    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs" pod "8b804fac-5e8d-4b4f-a194-d17214118b8b" (UID: "8b804fac-5e8d-4b4f-a194-d17214118b8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:21.229016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.228985    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm" pod "87554784-d0fe-4228-8dcf-c9a0af78a20f" (UID: "87554784-d0fe-4228-8dcf-c9a0af78a20f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.229016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.228992    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs" pod "87554784-d0fe-4228-8dcf-c9a0af78a20f" (UID: "87554784-d0fe-4228-8dcf-c9a0af78a20f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:21.229894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.229869    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm" pod "bcdefec2-1662-4bc5-b11b-308df189a01c" (UID: "bcdefec2-1662-4bc5-b11b-308df189a01c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:21.230366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.230345    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs" pod "bcdefec2-1662-4bc5-b11b-308df189a01c" (UID: "bcdefec2-1662-4bc5-b11b-308df189a01c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:21.230452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.230367    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv" pod "8b804fac-5e8d-4b4f-a194-d17214118b8b" (UID: "8b804fac-5e8d-4b4f-a194-d17214118b8b"). InnerVolumeSpecName "kube-api-access-lxxgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:21.236598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.236560    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74" pod "87554784-d0fe-4228-8dcf-c9a0af78a20f" (UID: "87554784-d0fe-4228-8dcf-c9a0af78a20f"). InnerVolumeSpecName "kube-api-access-vgq74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.326985    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxxgv\" (UniqueName: \"kubernetes.io/projected/8b804fac-5e8d-4b4f-a194-d17214118b8b-kube-api-access-lxxgv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327018    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdefec2-1662-4bc5-b11b-308df189a01c-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327031    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b804fac-5e8d-4b4f-a194-d17214118b8b-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327039    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm76j\" (UniqueName: \"kubernetes.io/projected/bcdefec2-1662-4bc5-b11b-308df189a01c-kube-api-access-nm76j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327047    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdefec2-1662-4bc5-b11b-308df189a01c-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327057    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87554784-d0fe-4228-8dcf-c9a0af78a20f-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327066    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327075    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgq74\" (UniqueName: \"kubernetes.io/projected/87554784-d0fe-4228-8dcf-c9a0af78a20f-kube-api-access-vgq74\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327085    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b804fac-5e8d-4b4f-a194-d17214118b8b-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.327490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.327092    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87554784-d0fe-4228-8dcf-c9a0af78a20f-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:21.481824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.481788    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:21.597857 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.597823    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:21.609931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.609896    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"]
Oct 30 18:06:21.613186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.613159    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-016f1d-vq4nv"]
Oct 30 18:06:21.713604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.713571    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:21.844611 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.844572    5534 generic.go:356] "Generic (PLEG): container finished" podID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerID="6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83" exitCode=0
Oct 30 18:06:21.844839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.844630    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"
Oct 30 18:06:21.844839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.844634    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" event={"ID":"8b804fac-5e8d-4b4f-a194-d17214118b8b","Type":"ContainerDied","Data":"6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83"}
Oct 30 18:06:21.844839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.844720    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729" event={"ID":"8b804fac-5e8d-4b4f-a194-d17214118b8b","Type":"ContainerDied","Data":"133720eb9b1a05ca2f88e5f8003037224a6479af6aa5693b9eb7aa36b43e7be2"}
Oct 30 18:06:21.844839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.844747    5534 scope.go:122] "RemoveContainer" containerID="6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83"
Oct 30 18:06:21.846316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.846294    5534 generic.go:356] "Generic (PLEG): container finished" podID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerID="e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2" exitCode=0
Oct 30 18:06:21.846405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.846343    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"
Oct 30 18:06:21.846405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.846351    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" event={"ID":"bcdefec2-1662-4bc5-b11b-308df189a01c","Type":"ContainerDied","Data":"e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2"}
Oct 30 18:06:21.846405 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.846388    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq" event={"ID":"bcdefec2-1662-4bc5-b11b-308df189a01c","Type":"ContainerDied","Data":"837d8d950395292133e234700a88f7c8d9011bacf8436b86c9ce74f618fc2cc7"}
Oct 30 18:06:21.848209 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.848144    5534 generic.go:356] "Generic (PLEG): container finished" podID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerID="59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7" exitCode=0
Oct 30 18:06:21.848359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.848328    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" event={"ID":"87554784-d0fe-4228-8dcf-c9a0af78a20f","Type":"ContainerDied","Data":"59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7"}
Oct 30 18:06:21.848426 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.848375    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp" event={"ID":"87554784-d0fe-4228-8dcf-c9a0af78a20f","Type":"ContainerDied","Data":"b2ef065a18dee30e92f40943f60633f80dd248028a8d9127a9647b669c7aebd3"}
Oct 30 18:06:21.848482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.848424    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"
Oct 30 18:06:21.859000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.858637    5534 scope.go:122] "RemoveContainer" containerID="6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83"
Oct 30 18:06:21.859513 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:21.859478    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83\": container with ID starting with 6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83 not found: ID does not exist" containerID="6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83"
Oct 30 18:06:21.859603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.859514    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83"} err="failed to get container status \"6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83\": rpc error: code = NotFound desc = could not find container \"6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83\": container with ID starting with 6212d53afe9a485a3cb8674930a3759cb028c0498d14c07682ac19ea38be0a83 not found: ID does not exist"
Oct 30 18:06:21.859603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.859537    5534 scope.go:122] "RemoveContainer" containerID="e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2"
Oct 30 18:06:21.871578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.871556    5534 scope.go:122] "RemoveContainer" containerID="e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2"
Oct 30 18:06:21.871952 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:21.871923    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2\": container with ID starting with e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2 not found: ID does not exist" containerID="e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2"
Oct 30 18:06:21.872032 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.871959    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2"} err="failed to get container status \"e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2\": rpc error: code = NotFound desc = could not find container \"e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2\": container with ID starting with e6f83af6ad43e40571ef0af9790c51899091e163648836ea64589423e64dd0c2 not found: ID does not exist"
Oct 30 18:06:21.872032 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.871977    5534 scope.go:122] "RemoveContainer" containerID="59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7"
Oct 30 18:06:21.882327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.882308    5534 scope.go:122] "RemoveContainer" containerID="59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7"
Oct 30 18:06:21.882669 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:21.882571    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7\": container with ID starting with 59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7 not found: ID does not exist" containerID="59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7"
Oct 30 18:06:21.882669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.882603    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7"} err="failed to get container status \"59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7\": rpc error: code = NotFound desc = could not find container \"59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7\": container with ID starting with 59edab9ae3cb60c68c9b5bb13d70099fa642b0324e2fcfe226caad3e7dcad2e7 not found: ID does not exist"
Oct 30 18:06:21.911136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.911112    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"]
Oct 30 18:06:21.922884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.922863    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-2-w5-db98b0-kserve-7c87b8bbdb-8rfxp"]
Oct 30 18:06:21.953571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.953539    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"]
Oct 30 18:06:21.959219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.959194    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/premium-w5-db98b0-kserve-7dd8c7c454-8m729"]
Oct 30 18:06:21.992306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.992276    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"]
Oct 30 18:06:21.996402 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:21.996381    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/simulator-w5-db98b0-kserve-85ffc4b59-p2zwq"]
Oct 30 18:06:22.125321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:22.125221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:22.370068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:22.370035    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466f653f-8036-49ce-82c7-d27fb7fb9804" path="/var/lib/kubelet/pods/466f653f-8036-49ce-82c7-d27fb7fb9804/volumes"
Oct 30 18:06:22.370453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:22.370440    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" path="/var/lib/kubelet/pods/87554784-d0fe-4228-8dcf-c9a0af78a20f/volumes"
Oct 30 18:06:22.370881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:22.370866    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" path="/var/lib/kubelet/pods/8b804fac-5e8d-4b4f-a194-d17214118b8b/volumes"
Oct 30 18:06:22.371524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:22.371510    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" path="/var/lib/kubelet/pods/bcdefec2-1662-4bc5-b11b-308df189a01c/volumes"
Oct 30 18:06:22.811187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:22.811158    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:06:23.599967 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:23.599926    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" podUID="841b5998-a749-4267-9df7-64ff045cc236" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:06:23.740833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:23.740788    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:24.299202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:24.299167    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:24.411086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:24.411056    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:24.426720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:24.426692    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"]
Oct 30 18:06:24.438601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:24.438576    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-455086-p6vkb"]
Oct 30 18:06:24.896531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:24.896493    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:25.039418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:25.039390    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:25.150004 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:25.149908    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:26.174475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:26.174435    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:26.371086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:26.371053    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc9d7d3-475a-480f-b0d1-6e138a46d1c3" path="/var/lib/kubelet/pods/acc9d7d3-475a-480f-b0d1-6e138a46d1c3/volumes"
Oct 30 18:06:27.876674 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:27.876618    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:27.988268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:27.988228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:28.111226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:28.111130    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:28.231385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:28.231300    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:28.432854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:28.432818    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"]
Oct 30 18:06:28.433533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:28.433497    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" containerID="cri-o://93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e" gracePeriod=30
Oct 30 18:06:28.467406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:28.467372    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:28.971404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:28.971369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:29.117835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.117795    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"]
Oct 30 18:06:29.118450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118433    5534 cpu_manager.go:414] "containerMap: removing container" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main"
Oct 30 18:06:29.118497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118453    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main"
Oct 30 18:06:29.118497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118481    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main"
Oct 30 18:06:29.118497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118490    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main"
Oct 30 18:06:29.118604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118499    5534 cpu_manager.go:414] "containerMap: removing container" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main"
Oct 30 18:06:29.118604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118505    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main"
Oct 30 18:06:29.118604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118523    5534 cpu_manager.go:414] "containerMap: removing container" podUID="679ed575-94a4-4801-a458-41f6c66d1631" containerName="revoke-keys"
Oct 30 18:06:29.118604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118529    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679ed575-94a4-4801-a458-41f6c66d1631" containerName="revoke-keys"
Oct 30 18:06:29.118604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118542    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerName="payload-processing"
Oct 30 18:06:29.118604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118547    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerName="payload-processing"
Oct 30 18:06:29.118843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118687    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="679ed575-94a4-4801-a458-41f6c66d1631" containerName="revoke-keys"
Oct 30 18:06:29.118843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118700    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2a645e-dbe2-4182-beff-2b3e9d59bab3" containerName="payload-processing"
Oct 30 18:06:29.118843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118711    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="87554784-d0fe-4228-8dcf-c9a0af78a20f" containerName="main"
Oct 30 18:06:29.118843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118720    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdefec2-1662-4bc5-b11b-308df189a01c" containerName="main"
Oct 30 18:06:29.118843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.118728    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b804fac-5e8d-4b4f-a194-d17214118b8b" containerName="main"
Oct 30 18:06:29.124026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.124006    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.143027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.142996    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"]
Oct 30 18:06:29.151200 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.151175    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:06:29.151375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.151228    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:06:29.275723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.275597    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.275906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.275765    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.275906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.275878    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pwr\" (UniqueName: \"kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.370487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.370453    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:29.377830 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.377796    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.377830 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.377832    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.378096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.377871    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pwr\" (UniqueName: \"kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.378529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.378505    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.380388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.380363    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.401469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.401431    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pwr\" (UniqueName: \"kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr\") pod \"maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.438361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.438325    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:29.440716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.440687    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:06:29.440856 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.440746    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:06:29.472630 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.472597    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"]
Oct 30 18:06:29.472893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.472863    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p" podUID="ce8c21f3-9028-4e39-a7e3-f4de43083a67" containerName="payload-processing" containerID="cri-o://a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab" gracePeriod=30
Oct 30 18:06:29.514597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.514559    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"]
Oct 30 18:06:29.514932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.514901    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr" podUID="b25abd8f-0f00-4e2e-a31d-9964689d83c2" containerName="payload-pre-processing" containerID="cri-o://ed26351f30c16fdfc7c295f6b4a62e4ad253904197fe8f7bd178dbd1aedcb9d0" gracePeriod=30
Oct 30 18:06:29.565553 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.565515    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:29.681690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.679740    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:29.852240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.852128    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"]
Oct 30 18:06:29.917116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.917020    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:06:29.928990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.928923    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx" event={"ID":"c1859fe4-820e-44cb-8bd8-82051ecddaca","Type":"ContainerStarted","Data":"a6caa0419004a3ef23b73df5b49d385fe641240694a4173f359a607d371e879a"}
Oct 30 18:06:29.930703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.930673    5534 generic.go:356] "Generic (PLEG): container finished" podID="ce8c21f3-9028-4e39-a7e3-f4de43083a67" containerID="a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab" exitCode=0
Oct 30 18:06:29.930828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.930787    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p" event={"ID":"ce8c21f3-9028-4e39-a7e3-f4de43083a67","Type":"ContainerDied","Data":"a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab"}
Oct 30 18:06:29.930905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.930831    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p" event={"ID":"ce8c21f3-9028-4e39-a7e3-f4de43083a67","Type":"ContainerDied","Data":"ae31bcff760d7e2ebf3f87d8fb2839626c14988db3048412f7243065e18b5026"}
Oct 30 18:06:29.930905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.930861    5534 scope.go:122] "RemoveContainer" containerID="a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab"
Oct 30 18:06:29.932056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.931704    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"
Oct 30 18:06:29.933906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.933791    5534 generic.go:356] "Generic (PLEG): container finished" podID="b25abd8f-0f00-4e2e-a31d-9964689d83c2" containerID="ed26351f30c16fdfc7c295f6b4a62e4ad253904197fe8f7bd178dbd1aedcb9d0" exitCode=0
Oct 30 18:06:29.933906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.933843    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr" event={"ID":"b25abd8f-0f00-4e2e-a31d-9964689d83c2","Type":"ContainerDied","Data":"ed26351f30c16fdfc7c295f6b4a62e4ad253904197fe8f7bd178dbd1aedcb9d0"}
Oct 30 18:06:29.948354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.948332    5534 scope.go:122] "RemoveContainer" containerID="a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab"
Oct 30 18:06:29.948863 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:29.948845    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab\": container with ID starting with a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab not found: ID does not exist" containerID="a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab"
Oct 30 18:06:29.949026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.949008    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab"} err="failed to get container status \"a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab\": rpc error: code = NotFound desc = could not find container \"a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab\": container with ID starting with a191d087263eb816ed6b08c1163628cf6eabb256ecc4c40f66ceff9fccc6e2ab not found: ID does not exist"
Oct 30 18:06:29.992653 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:29.992616    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:06:30.095908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.095877    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm\" (UniqueName: \"kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm\") pod \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") "
Oct 30 18:06:30.096074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.095998    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume\") pod \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") "
Oct 30 18:06:30.096074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.096055    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl\" (UniqueName: \"kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl\") pod \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\" (UID: \"b25abd8f-0f00-4e2e-a31d-9964689d83c2\") "
Oct 30 18:06:30.096216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.096200    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume\") pod \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\" (UID: \"ce8c21f3-9028-4e39-a7e3-f4de43083a67\") "
Oct 30 18:06:30.096471 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.096438    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume" pod "b25abd8f-0f00-4e2e-a31d-9964689d83c2" (UID: "b25abd8f-0f00-4e2e-a31d-9964689d83c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:30.096586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.096518    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume" pod "ce8c21f3-9028-4e39-a7e3-f4de43083a67" (UID: "ce8c21f3-9028-4e39-a7e3-f4de43083a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:30.096926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.096903    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8c21f3-9028-4e39-a7e3-f4de43083a67-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:30.097064 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.096931    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25abd8f-0f00-4e2e-a31d-9964689d83c2-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:30.098509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.098484    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm" pod "ce8c21f3-9028-4e39-a7e3-f4de43083a67" (UID: "ce8c21f3-9028-4e39-a7e3-f4de43083a67"). InnerVolumeSpecName "kube-api-access-mbbbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:30.098832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.098814    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl" pod "b25abd8f-0f00-4e2e-a31d-9964689d83c2" (UID: "b25abd8f-0f00-4e2e-a31d-9964689d83c2"). InnerVolumeSpecName "kube-api-access-vfddl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:30.152888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.152852    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:30.198484 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.198448    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbbm\" (UniqueName: \"kubernetes.io/projected/ce8c21f3-9028-4e39-a7e3-f4de43083a67-kube-api-access-mbbbm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:30.198484 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.198474    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfddl\" (UniqueName: \"kubernetes.io/projected/b25abd8f-0f00-4e2e-a31d-9964689d83c2-kube-api-access-vfddl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:30.267174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.267140    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:30.281078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.281046    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"]
Oct 30 18:06:30.285678 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.285632    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-745d5f58c6-6g64p"]
Oct 30 18:06:30.320721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.320688    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"]
Oct 30 18:06:30.324980 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.324950    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584a12-6ql9m"]
Oct 30 18:06:30.373924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.373839    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8c21f3-9028-4e39-a7e3-f4de43083a67" path="/var/lib/kubelet/pods/ce8c21f3-9028-4e39-a7e3-f4de43083a67/volumes"
Oct 30 18:06:30.374300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.374286    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6395d2-3393-4ad9-9148-b48fe0327add" path="/var/lib/kubelet/pods/ee6395d2-3393-4ad9-9148-b48fe0327add/volumes"
Oct 30 18:06:30.381813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.381781    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:30.828165 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.828133    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:30.944350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.944252    5534 generic.go:356] "Generic (PLEG): container finished" podID="c1859fe4-820e-44cb-8bd8-82051ecddaca" containerID="a35320ea7d8f194f65142e9ad038ab1bd41cf45be412cac80cc4506e1ff66a91" exitCode=0
Oct 30 18:06:30.944350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.944332    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx" event={"ID":"c1859fe4-820e-44cb-8bd8-82051ecddaca","Type":"ContainerDied","Data":"a35320ea7d8f194f65142e9ad038ab1bd41cf45be412cac80cc4506e1ff66a91"}
Oct 30 18:06:30.947177 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.947155    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"
Oct 30 18:06:30.947316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.947151    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr" event={"ID":"b25abd8f-0f00-4e2e-a31d-9964689d83c2","Type":"ContainerDied","Data":"4a757b2755566a364d0df87314944377f70d749c868029dd40e623c45e9f2792"}
Oct 30 18:06:30.947316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:30.947296    5534 scope.go:122] "RemoveContainer" containerID="ed26351f30c16fdfc7c295f6b4a62e4ad253904197fe8f7bd178dbd1aedcb9d0"
Oct 30 18:06:31.019801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.019743    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"]
Oct 30 18:06:31.023654 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.023617    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584a12-5787b68755-s9tlr"]
Oct 30 18:06:31.107810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.107770    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:31.218633 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.218499    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:31.420909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.420874    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"]
Oct 30 18:06:31.421508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.421477    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy" containerID="cri-o://29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd" gracePeriod=30
Oct 30 18:06:31.436292 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.436244    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:31.719370 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:31.719340    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:32.163926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.163898    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:32.190235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.190202    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:32.332512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.332421    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca\") pod \"c1859fe4-820e-44cb-8bd8-82051ecddaca\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") "
Oct 30 18:06:32.332512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.332470    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca\") pod \"c1859fe4-820e-44cb-8bd8-82051ecddaca\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") "
Oct 30 18:06:32.332512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.332512    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr\" (UniqueName: \"kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr\") pod \"c1859fe4-820e-44cb-8bd8-82051ecddaca\" (UID: \"c1859fe4-820e-44cb-8bd8-82051ecddaca\") "
Oct 30 18:06:32.332913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.332887    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca" pod "c1859fe4-820e-44cb-8bd8-82051ecddaca" (UID: "c1859fe4-820e-44cb-8bd8-82051ecddaca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:32.334709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.334689    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca" pod "c1859fe4-820e-44cb-8bd8-82051ecddaca" (UID: "c1859fe4-820e-44cb-8bd8-82051ecddaca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:32.334842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.334790    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr" pod "c1859fe4-820e-44cb-8bd8-82051ecddaca" (UID: "c1859fe4-820e-44cb-8bd8-82051ecddaca"). InnerVolumeSpecName "kube-api-access-p7pwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:32.370793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.370762    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25abd8f-0f00-4e2e-a31d-9964689d83c2" path="/var/lib/kubelet/pods/b25abd8f-0f00-4e2e-a31d-9964689d83c2/volumes"
Oct 30 18:06:32.433884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.433848    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:32.433884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.433878    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1859fe4-820e-44cb-8bd8-82051ecddaca-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:32.433884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.433888    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7pwr\" (UniqueName: \"kubernetes.io/projected/c1859fe4-820e-44cb-8bd8-82051ecddaca-kube-api-access-p7pwr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:32.480086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.480052    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:32.500542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.500497    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"]
Oct 30 18:06:32.504838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.504808    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-4dcdbb-dvkjv"]
Oct 30 18:06:32.598464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.598415    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:32.964304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.964213    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx" event={"ID":"c1859fe4-820e-44cb-8bd8-82051ecddaca","Type":"ContainerDied","Data":"a6caa0419004a3ef23b73df5b49d385fe641240694a4173f359a607d371e879a"}
Oct 30 18:06:32.964304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.964253    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"
Oct 30 18:06:32.964304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:32.964265    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6caa0419004a3ef23b73df5b49d385fe641240694a4173f359a607d371e879a"
Oct 30 18:06:33.000478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.000453    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:33.031442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.031413    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"]
Oct 30 18:06:33.034860 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.034840    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9879d034-7wmdd"]
Oct 30 18:06:33.600922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.600887    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" podUID="841b5998-a749-4267-9df7-64ff045cc236" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:06:33.734819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.734792    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 18:06:33.851151 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851118    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851175    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851194    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851216    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851253    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851325    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851379    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851452    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:33.851727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851475    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851532    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs\") pod \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\" (UID: \"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1\") "
Oct 30 18:06:33.851727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851597    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:33.852051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851816    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:33.852051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.851923    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:33.852051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.852005    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:33.852457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.852080    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.852457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.852101    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.852457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.852111    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.852457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.852120    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.852457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.852133    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.854167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.854143    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:33.854295 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.854249    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:06:33.855015 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.854989    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:33.855118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.855026    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm" pod "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" (UID: "ecc55c6d-f8ec-405c-82b3-6611f2ca26c1"). InnerVolumeSpecName "kube-api-access-x7btm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:33.927226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.927188    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"]
Oct 30 18:06:33.953246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.953217    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.953246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.953244    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7btm\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-kube-api-access-x7btm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.953246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.953256    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.953246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.953264    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:33.974090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.974016    5534 generic.go:356] "Generic (PLEG): container finished" podID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerID="93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e" exitCode=0
Oct 30 18:06:33.974090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.974072    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"
Oct 30 18:06:33.974267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.974088    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" event={"ID":"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1","Type":"ContainerDied","Data":"93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e"}
Oct 30 18:06:33.974267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.974138    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs" event={"ID":"ecc55c6d-f8ec-405c-82b3-6611f2ca26c1","Type":"ContainerDied","Data":"334d3e25fb4f396d6b7ed71897cdbed528b2d3ac25f322f9a9f4bee116af7c57"}
Oct 30 18:06:33.974267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.974169    5534 scope.go:122] "RemoveContainer" containerID="93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e"
Oct 30 18:06:33.992921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:33.992545    5534 scope.go:122] "RemoveContainer" containerID="db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"
Oct 30 18:06:34.003232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.003208    5534 scope.go:122] "RemoveContainer" containerID="93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e"
Oct 30 18:06:34.003526 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:34.003505    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e\": container with ID starting with 93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e not found: ID does not exist" containerID="93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e"
Oct 30 18:06:34.003581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.003535    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e"} err="failed to get container status \"93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e\": rpc error: code = NotFound desc = could not find container \"93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e\": container with ID starting with 93108937739dd080b2232d5e590525ca327467f5a2a364eddfa98a7f4055881e not found: ID does not exist"
Oct 30 18:06:34.003581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.003553    5534 scope.go:122] "RemoveContainer" containerID="db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"
Oct 30 18:06:34.003965 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:34.003939    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8\": container with ID starting with db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8 not found: ID does not exist" containerID="db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"
Oct 30 18:06:34.004058 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.003974    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8"} err="failed to get container status \"db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8\": rpc error: code = NotFound desc = could not find container \"db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8\": container with ID starting with db20afef4ffc25db0f17b06654f5384c69f9f4b9ee59d8ed299b0ae570aa35a8 not found: ID does not exist"
Oct 30 18:06:34.032408 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.032382    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:34.032628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.032610    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"]
Oct 30 18:06:34.036708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.036687    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-g79vs"]
Oct 30 18:06:34.371247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.371216    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0670fb6b-0681-455e-a46e-b9a18805a1d5" path="/var/lib/kubelet/pods/0670fb6b-0681-455e-a46e-b9a18805a1d5/volumes"
Oct 30 18:06:34.371790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.371767    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679ed575-94a4-4801-a458-41f6c66d1631" path="/var/lib/kubelet/pods/679ed575-94a4-4801-a458-41f6c66d1631/volumes"
Oct 30 18:06:34.372253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.372234    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" path="/var/lib/kubelet/pods/ecc55c6d-f8ec-405c-82b3-6611f2ca26c1/volumes"
Oct 30 18:06:34.494398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:34.494366    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:36.296676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.294736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:36.745618 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.745590    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 18:06:36.787578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.787548    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 18:06:36.798153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798123    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798181    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798241    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798280    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798317    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798366    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798451    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798510    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798604    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp\") pod \"62cfe7a4-8d40-41b6-a066-a6619d20138a\" (UID: \"62cfe7a4-8d40-41b6-a066-a6619d20138a\") "
Oct 30 18:06:36.798670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798605    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:36.799025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798901    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:36.799025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.798987    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:36.799252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.799191    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.799252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.799212    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62cfe7a4-8d40-41b6-a066-a6619d20138a-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.799252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.799227    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.799599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.799305    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:36.799599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.799358    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:36.800898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.800877    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:36.801071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.801054    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:06:36.801294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.801277    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:36.801402 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.801385    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp" pod "62cfe7a4-8d40-41b6-a066-a6619d20138a" (UID: "62cfe7a4-8d40-41b6-a066-a6619d20138a"). InnerVolumeSpecName "kube-api-access-skflp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:36.899855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.899820    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn\" (UniqueName: \"kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn\") pod \"841b5998-a749-4267-9df7-64ff045cc236\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") "
Oct 30 18:06:36.900071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.899900    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume\") pod \"841b5998-a749-4267-9df7-64ff045cc236\" (UID: \"841b5998-a749-4267-9df7-64ff045cc236\") "
Oct 30 18:06:36.900286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900262    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume" pod "841b5998-a749-4267-9df7-64ff045cc236" (UID: "841b5998-a749-4267-9df7-64ff045cc236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:36.900398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900381    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skflp\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-kube-api-access-skflp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.900461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900404    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.900461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900414    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.900461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900424    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.900461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900432    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.900461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900441    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b5998-a749-4267-9df7-64ff045cc236-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.900461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.900449    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62cfe7a4-8d40-41b6-a066-a6619d20138a-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:36.902679 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:36.902636    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn" pod "841b5998-a749-4267-9df7-64ff045cc236" (UID: "841b5998-a749-4267-9df7-64ff045cc236"). InnerVolumeSpecName "kube-api-access-fw8zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:37.000250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.000215    5534 generic.go:356] "Generic (PLEG): container finished" podID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerID="29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd" exitCode=0
Oct 30 18:06:37.000502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.000274    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"
Oct 30 18:06:37.000502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.000308    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" event={"ID":"62cfe7a4-8d40-41b6-a066-a6619d20138a","Type":"ContainerDied","Data":"29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd"}
Oct 30 18:06:37.000502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.000365    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz" event={"ID":"62cfe7a4-8d40-41b6-a066-a6619d20138a","Type":"ContainerDied","Data":"0c5c74eecb4ecff16d66f10af38afffd7451a47ba79ff1d0656aa2d6c794c421"}
Oct 30 18:06:37.000502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.000395    5534 scope.go:122] "RemoveContainer" containerID="29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd"
Oct 30 18:06:37.001269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.001240    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw8zn\" (UniqueName: \"kubernetes.io/projected/841b5998-a749-4267-9df7-64ff045cc236-kube-api-access-fw8zn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:37.002566 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.002538    5534 generic.go:356] "Generic (PLEG): container finished" podID="841b5998-a749-4267-9df7-64ff045cc236" containerID="c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15" exitCode=0
Oct 30 18:06:37.002699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.002673    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" event={"ID":"841b5998-a749-4267-9df7-64ff045cc236","Type":"ContainerDied","Data":"c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15"}
Oct 30 18:06:37.002783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.002694    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"
Oct 30 18:06:37.002783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.002725    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w" event={"ID":"841b5998-a749-4267-9df7-64ff045cc236","Type":"ContainerDied","Data":"0c71612ab58cd1fde3c69b29fc9b5811de77cc14d61779c9b9e1a23e6f549547"}
Oct 30 18:06:37.012768 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.012746    5534 scope.go:122] "RemoveContainer" containerID="e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"
Oct 30 18:06:37.023075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.023053    5534 scope.go:122] "RemoveContainer" containerID="29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd"
Oct 30 18:06:37.023427 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:37.023407    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd\": container with ID starting with 29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd not found: ID does not exist" containerID="29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd"
Oct 30 18:06:37.023524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.023437    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd"} err="failed to get container status \"29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd\": rpc error: code = NotFound desc = could not find container \"29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd\": container with ID starting with 29e84ed3281de681aee3593a84498429d21419ecc55575b801d66c27ccef02bd not found: ID does not exist"
Oct 30 18:06:37.023524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.023456    5534 scope.go:122] "RemoveContainer" containerID="e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"
Oct 30 18:06:37.023752 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:37.023728    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd\": container with ID starting with e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd not found: ID does not exist" containerID="e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"
Oct 30 18:06:37.023815 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.023761    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd"} err="failed to get container status \"e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd\": rpc error: code = NotFound desc = could not find container \"e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd\": container with ID starting with e5eca95370a2210d6d18eb92a4e42c0d926cd794ff65d5d4c39767a619b0eebd not found: ID does not exist"
Oct 30 18:06:37.023815 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.023782    5534 scope.go:122] "RemoveContainer" containerID="c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15"
Oct 30 18:06:37.034097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.034077    5534 scope.go:122] "RemoveContainer" containerID="c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15"
Oct 30 18:06:37.034411 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:37.034391    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15\": container with ID starting with c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15 not found: ID does not exist" containerID="c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15"
Oct 30 18:06:37.034464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.034419    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15"} err="failed to get container status \"c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15\": rpc error: code = NotFound desc = could not find container \"c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15\": container with ID starting with c642d1124bf505df4c1c9465f63702cf2d5ceca2218c10246c9517a86e9b2d15 not found: ID does not exist"
Oct 30 18:06:37.053199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.053160    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"]
Oct 30 18:06:37.057134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.057105    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7ffxz"]
Oct 30 18:06:37.095037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.095002    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"]
Oct 30 18:06:37.098940 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.098914    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db98b0-9c97b555-dx96w"]
Oct 30 18:06:37.183735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.183631    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:37.304988 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:37.304948    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:38.300392 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:38.300347    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:38.370529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:38.370494    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" path="/var/lib/kubelet/pods/62cfe7a4-8d40-41b6-a066-a6619d20138a/volumes"
Oct 30 18:06:38.371153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:38.371133    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841b5998-a749-4267-9df7-64ff045cc236" path="/var/lib/kubelet/pods/841b5998-a749-4267-9df7-64ff045cc236/volumes"
Oct 30 18:06:39.314258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:39.314229    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:39.439956 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:39.439921    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:39.608115 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:39.608076    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"]
Oct 30 18:06:39.695801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:39.695444    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:39.807436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:39.807401    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:40.039657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.039565    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:40.173667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.169919    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:40.325250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325171    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"]
Oct 30 18:06:40.325836 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325822    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy"
Oct 30 18:06:40.325889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325838    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy"
Oct 30 18:06:40.325889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325852    5534 cpu_manager.go:414] "containerMap: removing container" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy"
Oct 30 18:06:40.325889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325857    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy"
Oct 30 18:06:40.325889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325871    5534 cpu_manager.go:414] "containerMap: removing container" podUID="841b5998-a749-4267-9df7-64ff045cc236" containerName="payload-pre-processing"
Oct 30 18:06:40.325889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325876    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841b5998-a749-4267-9df7-64ff045cc236" containerName="payload-pre-processing"
Oct 30 18:06:40.325889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325890    5534 cpu_manager.go:414] "containerMap: removing container" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325895    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325903    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b25abd8f-0f00-4e2e-a31d-9964689d83c2" containerName="payload-pre-processing"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325907    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25abd8f-0f00-4e2e-a31d-9964689d83c2" containerName="payload-pre-processing"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325923    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c1859fe4-820e-44cb-8bd8-82051ecddaca" containerName="revoke-keys"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325927    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1859fe4-820e-44cb-8bd8-82051ecddaca" containerName="revoke-keys"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325935    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ce8c21f3-9028-4e39-a7e3-f4de43083a67" containerName="payload-processing"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.325939    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8c21f3-9028-4e39-a7e3-f4de43083a67" containerName="payload-processing"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326031    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326041    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326051    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="841b5998-a749-4267-9df7-64ff045cc236" containerName="payload-pre-processing"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326056    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cfe7a4-8d40-41b6-a066-a6619d20138a" containerName="istio-proxy"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326062    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25abd8f-0f00-4e2e-a31d-9964689d83c2" containerName="payload-pre-processing"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326069    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1859fe4-820e-44cb-8bd8-82051ecddaca" containerName="revoke-keys"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326074    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy"
Oct 30 18:06:40.326076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.326081    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8c21f3-9028-4e39-a7e3-f4de43083a67" containerName="payload-processing"
Oct 30 18:06:40.331357 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.331336    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.349535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.349504    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"]
Oct 30 18:06:40.423796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.423770    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:06:40.423974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.423808    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:06:40.471830 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.471797    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.472284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.471977    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.472284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.472018    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlphl\" (UniqueName: \"kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.573756 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.573719    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.573948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.573809    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.573948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.573831    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlphl\" (UniqueName: \"kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.574415 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.574394    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.576527 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.576474    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.597379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.597351    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlphl\" (UniqueName: \"kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl\") pod \"maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.646034 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.645992    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:40.981046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:40.979886    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"]
Oct 30 18:06:40.981795 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:06:40.981767    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635c72fe_c81e_4b64_8f39_5077c220517c.slice/crio-1c5b609826ef44b0edcf31b3394802ae94e130a5914009ebd2d5e2eccebfd53a WatchSource:0}: Error finding container 1c5b609826ef44b0edcf31b3394802ae94e130a5914009ebd2d5e2eccebfd53a: Status 404 returned error can't find the container with id 1c5b609826ef44b0edcf31b3394802ae94e130a5914009ebd2d5e2eccebfd53a
Oct 30 18:06:41.036653 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:41.036607    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf" event={"ID":"635c72fe-c81e-4b64-8f39-5077c220517c","Type":"ContainerStarted","Data":"1c5b609826ef44b0edcf31b3394802ae94e130a5914009ebd2d5e2eccebfd53a"}
Oct 30 18:06:41.817531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:41.817408    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:42.046576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:42.046535    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf" event={"ID":"635c72fe-c81e-4b64-8f39-5077c220517c","Type":"ContainerStarted","Data":"730c4a7f866b192a2100316d3a70e13d68e6db29f137aecc71fb9b2139147393"}
Oct 30 18:06:42.088127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:42.088052    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf" podStartSLOduration=2.088031781 podStartE2EDuration="2.088031781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:42.085656369 +0000 UTC m=+3207.847700174" watchObservedRunningTime="2026-10-30 18:06:42.088031781 +0000 UTC m=+3207.850075595"
Oct 30 18:06:42.373676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:42.369409    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:42.483987 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:42.483904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:43.056493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:43.056459    5534 generic.go:356] "Generic (PLEG): container finished" podID="635c72fe-c81e-4b64-8f39-5077c220517c" containerID="730c4a7f866b192a2100316d3a70e13d68e6db29f137aecc71fb9b2139147393" exitCode=0
Oct 30 18:06:43.057006 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:43.056554    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf" event={"ID":"635c72fe-c81e-4b64-8f39-5077c220517c","Type":"ContainerDied","Data":"730c4a7f866b192a2100316d3a70e13d68e6db29f137aecc71fb9b2139147393"}
Oct 30 18:06:43.820540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:43.820498    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"]
Oct 30 18:06:43.850097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:43.850061    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"]
Oct 30 18:06:43.949239 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:43.949183    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:44.029259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.029229    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"]
Oct 30 18:06:44.033430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.033398    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38255b35-cmj8v"]
Oct 30 18:06:44.085375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.085253    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:44.216699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.216551    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:44.320454 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.320428    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:44.393065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.393034    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebb47eb-d9bb-4147-9b45-3e24bf5058dd" path="/var/lib/kubelet/pods/6ebb47eb-d9bb-4147-9b45-3e24bf5058dd/volumes"
Oct 30 18:06:44.439190 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.439152    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca\") pod \"635c72fe-c81e-4b64-8f39-5077c220517c\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") "
Oct 30 18:06:44.439373 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.439211    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl\" (UniqueName: \"kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl\") pod \"635c72fe-c81e-4b64-8f39-5077c220517c\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") "
Oct 30 18:06:44.439373 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.439287    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca\") pod \"635c72fe-c81e-4b64-8f39-5077c220517c\" (UID: \"635c72fe-c81e-4b64-8f39-5077c220517c\") "
Oct 30 18:06:44.440792 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.440612    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca" pod "635c72fe-c81e-4b64-8f39-5077c220517c" (UID: "635c72fe-c81e-4b64-8f39-5077c220517c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:44.442902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.442867    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca" pod "635c72fe-c81e-4b64-8f39-5077c220517c" (UID: "635c72fe-c81e-4b64-8f39-5077c220517c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:44.490673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.464720    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl" pod "635c72fe-c81e-4b64-8f39-5077c220517c" (UID: "635c72fe-c81e-4b64-8f39-5077c220517c"). InnerVolumeSpecName "kube-api-access-xlphl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:44.540750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.540723    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:44.540750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.540746    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlphl\" (UniqueName: \"kubernetes.io/projected/635c72fe-c81e-4b64-8f39-5077c220517c-kube-api-access-xlphl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:44.540750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.540756    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635c72fe-c81e-4b64-8f39-5077c220517c-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:44.608680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:44.608632    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" containerName="payload-pre-processing" containerID="cri-o://62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00" gracePeriod=25
Oct 30 18:06:45.101790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.101759    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"
Oct 30 18:06:45.102244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.101788    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf" event={"ID":"635c72fe-c81e-4b64-8f39-5077c220517c","Type":"ContainerDied","Data":"1c5b609826ef44b0edcf31b3394802ae94e130a5914009ebd2d5e2eccebfd53a"}
Oct 30 18:06:45.102244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.101856    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5b609826ef44b0edcf31b3394802ae94e130a5914009ebd2d5e2eccebfd53a"
Oct 30 18:06:45.151057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.151026    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"]
Oct 30 18:06:45.233326 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.233291    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:45.247592 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:45.247555    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:06:45.345614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.345578    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:45.936600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:45.936553    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:46.050198 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:46.050159    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:47.530049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:47.530014    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:48.013905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.013865    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"]
Oct 30 18:06:48.014541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.014520    5534 cpu_manager.go:414] "containerMap: removing container" podUID="635c72fe-c81e-4b64-8f39-5077c220517c" containerName="revoke-keys"
Oct 30 18:06:48.014697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.014544    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635c72fe-c81e-4b64-8f39-5077c220517c" containerName="revoke-keys"
Oct 30 18:06:48.014697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.014558    5534 cpu_manager.go:414] "containerMap: removing container" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy"
Oct 30 18:06:48.014697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.014566    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc55c6d-f8ec-405c-82b3-6611f2ca26c1" containerName="istio-proxy"
Oct 30 18:06:48.014896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.014742    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="635c72fe-c81e-4b64-8f39-5077c220517c" containerName="revoke-keys"
Oct 30 18:06:48.019127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.019105    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.038359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.037969    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"]
Oct 30 18:06:48.045817 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.045787    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:06:48.045994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.045869    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:06:48.121839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.121804    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8mc\" (UniqueName: \"kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.122228 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.122201    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.122393 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.122376    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.215407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.215331    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:48.225016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.224140    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8mc\" (UniqueName: \"kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.225016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.224223    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.225016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.224249    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.225016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.224950    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.227314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.227291    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.287745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.262391    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8mc\" (UniqueName: \"kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc\") pod \"maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.337984 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.337953    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:48.693928 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.691259    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"]
Oct 30 18:06:48.696943 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:06:48.696914    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2562b78e_3f04_4820_9bbc_7d0399849c16.slice/crio-c958b1d580bf520874cb1ebf512311b3f84f2018fd90837a287a777a4f32bebb WatchSource:0}: Error finding container c958b1d580bf520874cb1ebf512311b3f84f2018fd90837a287a777a4f32bebb: Status 404 returned error can't find the container with id c958b1d580bf520874cb1ebf512311b3f84f2018fd90837a287a777a4f32bebb
Oct 30 18:06:48.821228 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.821184    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z" podUID="7db330d8-6e15-4eda-8832-346989df4ecf" containerName="payload-processing" containerID="cri-o://6ef3e50532f80933bba36b9f1d2f32df50f1709893506bcb16979583bf82bf12" gracePeriod=25
Oct 30 18:06:48.851466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.851286    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr" podUID="cbf26a2c-10f9-4016-972c-3e7708beb128" containerName="payload-pre-processing" containerID="cri-o://b46a7dab69153bd851a2c4f3f6665a75cbae5b797c5df3c500e088014ca87a68" gracePeriod=25
Oct 30 18:06:48.949910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.949861    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" containerID="cri-o://145cb8965faf903610e39a27063de1f1f744f8765e00e9a8d78eba36e46f009f" gracePeriod=45
Oct 30 18:06:48.995877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:48.994986    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": read tcp 10.133.0.2:33624->10.133.0.49:8000: read: connection reset by peer"
Oct 30 18:06:49.015561 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:49.015470    5534 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7bdf7e9_e0e6_4865_927e_418b7a7bc236.slice/crio-conmon-145cb8965faf903610e39a27063de1f1f744f8765e00e9a8d78eba36e46f009f.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:06:49.148575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.148454    5534 generic.go:356] "Generic (PLEG): container finished" podID="cbf26a2c-10f9-4016-972c-3e7708beb128" containerID="b46a7dab69153bd851a2c4f3f6665a75cbae5b797c5df3c500e088014ca87a68" exitCode=0
Oct 30 18:06:49.148575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.148519    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr" event={"ID":"cbf26a2c-10f9-4016-972c-3e7708beb128","Type":"ContainerDied","Data":"b46a7dab69153bd851a2c4f3f6665a75cbae5b797c5df3c500e088014ca87a68"}
Oct 30 18:06:49.151699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.150906    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc" event={"ID":"2562b78e-3f04-4820-9bbc-7d0399849c16","Type":"ContainerStarted","Data":"fdd208a076c1352c2addd7ceec51ea27c157a24cd05b92d97d7a29ebf739be7e"}
Oct 30 18:06:49.151699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.150952    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc" event={"ID":"2562b78e-3f04-4820-9bbc-7d0399849c16","Type":"ContainerStarted","Data":"c958b1d580bf520874cb1ebf512311b3f84f2018fd90837a287a777a4f32bebb"}
Oct 30 18:06:49.164676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.153332    5534 generic.go:356] "Generic (PLEG): container finished" podID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerID="145cb8965faf903610e39a27063de1f1f744f8765e00e9a8d78eba36e46f009f" exitCode=0
Oct 30 18:06:49.164676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.153429    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" event={"ID":"e7bdf7e9-e0e6-4865-927e-418b7a7bc236","Type":"ContainerDied","Data":"145cb8965faf903610e39a27063de1f1f744f8765e00e9a8d78eba36e46f009f"}
Oct 30 18:06:49.164676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.155124    5534 generic.go:356] "Generic (PLEG): container finished" podID="7db330d8-6e15-4eda-8832-346989df4ecf" containerID="6ef3e50532f80933bba36b9f1d2f32df50f1709893506bcb16979583bf82bf12" exitCode=0
Oct 30 18:06:49.164676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.155207    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z" event={"ID":"7db330d8-6e15-4eda-8832-346989df4ecf","Type":"ContainerDied","Data":"6ef3e50532f80933bba36b9f1d2f32df50f1709893506bcb16979583bf82bf12"}
Oct 30 18:06:49.193956 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.193854    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:06:49.197955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.197906    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc" podStartSLOduration=2.197889959 podStartE2EDuration="2.197889959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:49.191469796 +0000 UTC m=+3214.953513610" watchObservedRunningTime="2026-10-30 18:06:49.197889959 +0000 UTC m=+3214.959933772"
Oct 30 18:06:49.292899 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.292873    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:06:49.345774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.344732    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8\" (UniqueName: \"kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8\") pod \"7db330d8-6e15-4eda-8832-346989df4ecf\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") "
Oct 30 18:06:49.345774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.344789    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume\") pod \"7db330d8-6e15-4eda-8832-346989df4ecf\" (UID: \"7db330d8-6e15-4eda-8832-346989df4ecf\") "
Oct 30 18:06:49.345774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.345559    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume" pod "7db330d8-6e15-4eda-8832-346989df4ecf" (UID: "7db330d8-6e15-4eda-8832-346989df4ecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:49.352783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.352747    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8" pod "7db330d8-6e15-4eda-8832-346989df4ecf" (UID: "7db330d8-6e15-4eda-8832-346989df4ecf"). InnerVolumeSpecName "kube-api-access-w2kk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:49.390043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.390018    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 18:06:49.448399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.448294    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f\" (UniqueName: \"kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f\") pod \"cbf26a2c-10f9-4016-972c-3e7708beb128\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") "
Oct 30 18:06:49.448399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.448361    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume\") pod \"cbf26a2c-10f9-4016-972c-3e7708beb128\" (UID: \"cbf26a2c-10f9-4016-972c-3e7708beb128\") "
Oct 30 18:06:49.449313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.449257    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume" pod "cbf26a2c-10f9-4016-972c-3e7708beb128" (UID: "cbf26a2c-10f9-4016-972c-3e7708beb128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:49.451170 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.451142    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f" pod "cbf26a2c-10f9-4016-972c-3e7708beb128" (UID: "cbf26a2c-10f9-4016-972c-3e7708beb128"). InnerVolumeSpecName "kube-api-access-gts2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:49.452225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.452204    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2kk8\" (UniqueName: \"kubernetes.io/projected/7db330d8-6e15-4eda-8832-346989df4ecf-kube-api-access-w2kk8\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.452304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.452230    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db330d8-6e15-4eda-8832-346989df4ecf-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.452304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.452245    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gts2f\" (UniqueName: \"kubernetes.io/projected/cbf26a2c-10f9-4016-972c-3e7708beb128-kube-api-access-gts2f\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.452304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.452259    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf26a2c-10f9-4016-972c-3e7708beb128-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.553672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.553611    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache\") pod \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") "
Oct 30 18:06:49.553844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.553687    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs\") pod \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") "
Oct 30 18:06:49.553844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.553744    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir\") pod \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") "
Oct 30 18:06:49.553844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.553790    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home\") pod \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") "
Oct 30 18:06:49.554036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.553855    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm\") pod \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") "
Oct 30 18:06:49.554036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.553923    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache" pod "e7bdf7e9-e0e6-4865-927e-418b7a7bc236" (UID: "e7bdf7e9-e0e6-4865-927e-418b7a7bc236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:49.554036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.554002    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd\" (UniqueName: \"kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd\") pod \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\" (UID: \"e7bdf7e9-e0e6-4865-927e-418b7a7bc236\") "
Oct 30 18:06:49.554209 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.554145    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir" pod "e7bdf7e9-e0e6-4865-927e-418b7a7bc236" (UID: "e7bdf7e9-e0e6-4865-927e-418b7a7bc236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:49.554315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.554294    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home" pod "e7bdf7e9-e0e6-4865-927e-418b7a7bc236" (UID: "e7bdf7e9-e0e6-4865-927e-418b7a7bc236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:49.555016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.554991    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.555154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.555021    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.555154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.555036    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.557168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.557139    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd" pod "e7bdf7e9-e0e6-4865-927e-418b7a7bc236" (UID: "e7bdf7e9-e0e6-4865-927e-418b7a7bc236"). InnerVolumeSpecName "kube-api-access-rvrbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:49.557281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.557243    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs" pod "e7bdf7e9-e0e6-4865-927e-418b7a7bc236" (UID: "e7bdf7e9-e0e6-4865-927e-418b7a7bc236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:49.557281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.557260    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm" pod "e7bdf7e9-e0e6-4865-927e-418b7a7bc236" (UID: "e7bdf7e9-e0e6-4865-927e-418b7a7bc236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:06:49.657094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.657060    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.657094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.657097    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrbd\" (UniqueName: \"kubernetes.io/projected/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-kube-api-access-rvrbd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.657385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.657114    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bdf7e9-e0e6-4865-927e-418b7a7bc236-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:49.814899 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.814862    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"]
Oct 30 18:06:49.815772 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815748    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cbf26a2c-10f9-4016-972c-3e7708beb128" containerName="payload-pre-processing"
Oct 30 18:06:49.815831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815775    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf26a2c-10f9-4016-972c-3e7708beb128" containerName="payload-pre-processing"
Oct 30 18:06:49.815831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815794    5534 cpu_manager.go:414] "containerMap: removing container" podUID="7db330d8-6e15-4eda-8832-346989df4ecf" containerName="payload-processing"
Oct 30 18:06:49.815831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815803    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db330d8-6e15-4eda-8832-346989df4ecf" containerName="payload-processing"
Oct 30 18:06:49.815831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815821    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main"
Oct 30 18:06:49.815831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815830    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main"
Oct 30 18:06:49.816042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.815994    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf26a2c-10f9-4016-972c-3e7708beb128" containerName="payload-pre-processing"
Oct 30 18:06:49.816042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.816007    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db330d8-6e15-4eda-8832-346989df4ecf" containerName="payload-processing"
Oct 30 18:06:49.816042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.816019    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" containerName="main"
Oct 30 18:06:49.827445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.822180    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.844093 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.842603    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"]
Oct 30 18:06:49.859636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.859597    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.859953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.859932    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp5t\" (UniqueName: \"kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.860159 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.860143    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.961544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.961508    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.961982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.961956    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.962154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.962130    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp5t\" (UniqueName: \"kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.962620 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.962597    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.964701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.964678    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:49.985212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:49.985185    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp5t\" (UniqueName: \"kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t\") pod \"maas-api-revoke-sub-3dccee96-k8b7s\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:50.149840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.149804    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:50.178850 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.178813    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr" event={"ID":"cbf26a2c-10f9-4016-972c-3e7708beb128","Type":"ContainerDied","Data":"88891ec3348aec677ff6c0beee189af35bb2fd6285abf5923878bb5d7ceb24b4"}
Oct 30 18:06:50.179005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.178890    5534 scope.go:122] "RemoveContainer" containerID="b46a7dab69153bd851a2c4f3f6665a75cbae5b797c5df3c500e088014ca87a68"
Oct 30 18:06:50.179077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.179064    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"
Oct 30 18:06:50.184019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.183987    5534 generic.go:356] "Generic (PLEG): container finished" podID="2562b78e-3f04-4820-9bbc-7d0399849c16" containerID="fdd208a076c1352c2addd7ceec51ea27c157a24cd05b92d97d7a29ebf739be7e" exitCode=0
Oct 30 18:06:50.184150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.184100    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc" event={"ID":"2562b78e-3f04-4820-9bbc-7d0399849c16","Type":"ContainerDied","Data":"fdd208a076c1352c2addd7ceec51ea27c157a24cd05b92d97d7a29ebf739be7e"}
Oct 30 18:06:50.188452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.185867    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t" event={"ID":"e7bdf7e9-e0e6-4865-927e-418b7a7bc236","Type":"ContainerDied","Data":"f4a70d98285bb716c8bfc5430036b9c20f9f4299fbc673a00ac0fc48dc040b6d"}
Oct 30 18:06:50.188452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.185962    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"
Oct 30 18:06:50.192932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.192884    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z" event={"ID":"7db330d8-6e15-4eda-8832-346989df4ecf","Type":"ContainerDied","Data":"fd9f63e55f62276ce6ac4dda6d109a42c5ceb34a73557c7eda674a09054a01f9"}
Oct 30 18:06:50.193086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.193069    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"
Oct 30 18:06:50.199831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.199337    5534 scope.go:122] "RemoveContainer" containerID="145cb8965faf903610e39a27063de1f1f744f8765e00e9a8d78eba36e46f009f"
Oct 30 18:06:50.216450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.216431    5534 scope.go:122] "RemoveContainer" containerID="6ef3e50532f80933bba36b9f1d2f32df50f1709893506bcb16979583bf82bf12"
Oct 30 18:06:50.261578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.261546    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:50.318943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.315238    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"]
Oct 30 18:06:50.322655 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.322487    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-85b46c56fb-z492z"]
Oct 30 18:06:50.351724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.351679    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"]
Oct 30 18:06:50.354352 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.354322    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-506124/premium-w2-506124-kserve-6cc4754cc4-rgv9t"]
Oct 30 18:06:50.375898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.375052    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db330d8-6e15-4eda-8832-346989df4ecf" path="/var/lib/kubelet/pods/7db330d8-6e15-4eda-8832-346989df4ecf/volumes"
Oct 30 18:06:50.375898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.375812    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bdf7e9-e0e6-4865-927e-418b7a7bc236" path="/var/lib/kubelet/pods/e7bdf7e9-e0e6-4865-927e-418b7a7bc236/volumes"
Oct 30 18:06:50.390943 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.390911    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"]
Oct 30 18:06:50.394138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.394106    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-85d45c7c5-p76hr"]
Oct 30 18:06:50.509776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.509738    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"]
Oct 30 18:06:50.512302 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:06:50.512267    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b072561_7419_4119_a650_202284e75d91.slice/crio-af60d9bafba686044608d4b8f4aba84644c253d5ea62b5373c68e0ba155c1f9e WatchSource:0}: Error finding container af60d9bafba686044608d4b8f4aba84644c253d5ea62b5373c68e0ba155c1f9e: Status 404 returned error can't find the container with id af60d9bafba686044608d4b8f4aba84644c253d5ea62b5373c68e0ba155c1f9e
Oct 30 18:06:50.792717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.792674    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"]
Oct 30 18:06:50.822462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.822422    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"]
Oct 30 18:06:50.989755 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:50.989718    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:51.103843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.103812    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:51.206080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.206037    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s" event={"ID":"9b072561-7419-4119-a650-202284e75d91","Type":"ContainerStarted","Data":"149322fb854dfcba965182e09d2e93c92da053e6d20b4fc8471b58eab0f6bf64"}
Oct 30 18:06:51.206271 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.206101    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s" event={"ID":"9b072561-7419-4119-a650-202284e75d91","Type":"ContainerStarted","Data":"af60d9bafba686044608d4b8f4aba84644c253d5ea62b5373c68e0ba155c1f9e"}
Oct 30 18:06:51.249446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.249254    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s" podStartSLOduration=2.249229169 podStartE2EDuration="2.249229169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:51.245394042 +0000 UTC m=+3217.007437867" watchObservedRunningTime="2026-10-30 18:06:51.249229169 +0000 UTC m=+3217.011272983"
Oct 30 18:06:51.451268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.451236    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:51.508423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.508382    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca\") pod \"2562b78e-3f04-4820-9bbc-7d0399849c16\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") "
Oct 30 18:06:51.508621 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.508475    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc\" (UniqueName: \"kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc\") pod \"2562b78e-3f04-4820-9bbc-7d0399849c16\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") "
Oct 30 18:06:51.508621 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.508505    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca\") pod \"2562b78e-3f04-4820-9bbc-7d0399849c16\" (UID: \"2562b78e-3f04-4820-9bbc-7d0399849c16\") "
Oct 30 18:06:51.509245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.509219    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca" pod "2562b78e-3f04-4820-9bbc-7d0399849c16" (UID: "2562b78e-3f04-4820-9bbc-7d0399849c16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:51.553397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.553169    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc" pod "2562b78e-3f04-4820-9bbc-7d0399849c16" (UID: "2562b78e-3f04-4820-9bbc-7d0399849c16"). InnerVolumeSpecName "kube-api-access-8r8mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:51.553397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.553202    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca" pod "2562b78e-3f04-4820-9bbc-7d0399849c16" (UID: "2562b78e-3f04-4820-9bbc-7d0399849c16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:51.609814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.609776    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:51.609814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.609807    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8mc\" (UniqueName: \"kubernetes.io/projected/2562b78e-3f04-4820-9bbc-7d0399849c16-kube-api-access-8r8mc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:51.609814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.609817    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562b78e-3f04-4820-9bbc-7d0399849c16-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:51.885457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:51.885417    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:52.030025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.029984    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:52.149526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.149442    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:52.216777 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.216740    5534 generic.go:356] "Generic (PLEG): container finished" podID="9b072561-7419-4119-a650-202284e75d91" containerID="149322fb854dfcba965182e09d2e93c92da053e6d20b4fc8471b58eab0f6bf64" exitCode=0
Oct 30 18:06:52.216958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.216826    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s" event={"ID":"9b072561-7419-4119-a650-202284e75d91","Type":"ContainerDied","Data":"149322fb854dfcba965182e09d2e93c92da053e6d20b4fc8471b58eab0f6bf64"}
Oct 30 18:06:52.218314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.218291    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"
Oct 30 18:06:52.218422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.218325    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc" event={"ID":"2562b78e-3f04-4820-9bbc-7d0399849c16","Type":"ContainerDied","Data":"c958b1d580bf520874cb1ebf512311b3f84f2018fd90837a287a777a4f32bebb"}
Oct 30 18:06:52.218422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.218370    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c958b1d580bf520874cb1ebf512311b3f84f2018fd90837a287a777a4f32bebb"
Oct 30 18:06:52.371915 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:52.371878    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf26a2c-10f9-4016-972c-3e7708beb128" path="/var/lib/kubelet/pods/cbf26a2c-10f9-4016-972c-3e7708beb128/volumes"
Oct 30 18:06:53.227234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.227139    5534 generic.go:356] "Generic (PLEG): container finished" podID="9355dd9f-4e3b-4584-a332-068331172693" containerID="67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7" exitCode=0
Oct 30 18:06:53.227234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.227220    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" event={"ID":"9355dd9f-4e3b-4584-a332-068331172693","Type":"ContainerDied","Data":"67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"}
Oct 30 18:06:53.227887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.227783    5534 scope.go:122] "RemoveContainer" containerID="67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"
Oct 30 18:06:53.229216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.229189    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:06:53.229325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.229253    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:06:53.229325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.229287    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:06:53.320662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.320616    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:53.451869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.451839    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:53.531292 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.531250    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t\" (UniqueName: \"kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t\") pod \"9b072561-7419-4119-a650-202284e75d91\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") "
Oct 30 18:06:53.531559 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.531540    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca\") pod \"9b072561-7419-4119-a650-202284e75d91\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") "
Oct 30 18:06:53.531729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.531713    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca\") pod \"9b072561-7419-4119-a650-202284e75d91\" (UID: \"9b072561-7419-4119-a650-202284e75d91\") "
Oct 30 18:06:53.533579 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.533308    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca" pod "9b072561-7419-4119-a650-202284e75d91" (UID: "9b072561-7419-4119-a650-202284e75d91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:53.536220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.536176    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t" pod "9b072561-7419-4119-a650-202284e75d91" (UID: "9b072561-7419-4119-a650-202284e75d91"). InnerVolumeSpecName "kube-api-access-pmp5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:53.538261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.538240    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca" pod "9b072561-7419-4119-a650-202284e75d91" (UID: "9b072561-7419-4119-a650-202284e75d91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:06:53.642664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.637550    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp5t\" (UniqueName: \"kubernetes.io/projected/9b072561-7419-4119-a650-202284e75d91-kube-api-access-pmp5t\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:53.642664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.637600    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b072561-7419-4119-a650-202284e75d91-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:53.642664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.637616    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b072561-7419-4119-a650-202284e75d91-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:53.849601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.849565    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:53.956258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.956171    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"]
Oct 30 18:06:53.959510 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.959481    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db98b0-94vkb"]
Oct 30 18:06:53.960748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:53.960719    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:54.078726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.078692    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:54.235096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.235013    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"
Oct 30 18:06:54.235096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.235027    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s" event={"ID":"9b072561-7419-4119-a650-202284e75d91","Type":"ContainerDied","Data":"af60d9bafba686044608d4b8f4aba84644c253d5ea62b5373c68e0ba155c1f9e"}
Oct 30 18:06:54.235727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.235111    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af60d9bafba686044608d4b8f4aba84644c253d5ea62b5373c68e0ba155c1f9e"
Oct 30 18:06:54.237228 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.237196    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" event={"ID":"9355dd9f-4e3b-4584-a332-068331172693","Type":"ContainerStarted","Data":"ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a"}
Oct 30 18:06:54.261812 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.261695    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:54.384725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.384634    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffdd6ef-dbf0-4515-b509-c216c7de4882" path="/var/lib/kubelet/pods/1ffdd6ef-dbf0-4515-b509-c216c7de4882/volumes"
Oct 30 18:06:54.419567 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.419522    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:06:54.421145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.421113    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body=
Oct 30 18:06:54.421294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.421178    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused"
Oct 30 18:06:54.487225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.487097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:54.607151 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:54.607116    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:06:55.419680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:55.419545    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body=
Oct 30 18:06:55.419680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:55.419611    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused"
Oct 30 18:06:55.793056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:55.792958    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz" podUID="b0d64771-f9aa-462c-bd01-2fa4d430d7df" containerName="payload-processing" containerID="cri-o://381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9" gracePeriod=25
Oct 30 18:06:55.823711 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:55.823634    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg" podUID="82557d68-0eee-480b-960b-90087e394d52" containerName="payload-pre-processing" containerID="cri-o://b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263" gracePeriod=25
Oct 30 18:06:56.146998 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.146502    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:56.170681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.162828    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 18:06:56.199460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.194150    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume\") pod \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") "
Oct 30 18:06:56.199460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.194335    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv\" (UniqueName: \"kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv\") pod \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\" (UID: \"b0d64771-f9aa-462c-bd01-2fa4d430d7df\") "
Oct 30 18:06:56.201547 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.201483    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume" pod "b0d64771-f9aa-462c-bd01-2fa4d430d7df" (UID: "b0d64771-f9aa-462c-bd01-2fa4d430d7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:56.204745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.204711    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv" pod "b0d64771-f9aa-462c-bd01-2fa4d430d7df" (UID: "b0d64771-f9aa-462c-bd01-2fa4d430d7df"). InnerVolumeSpecName "kube-api-access-dpgwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:56.225208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.225151    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 18:06:56.289504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.289397    5534 generic.go:356] "Generic (PLEG): container finished" podID="b0d64771-f9aa-462c-bd01-2fa4d430d7df" containerID="381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9" exitCode=0
Oct 30 18:06:56.289504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.289467    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"
Oct 30 18:06:56.289824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.289438    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz" event={"ID":"b0d64771-f9aa-462c-bd01-2fa4d430d7df","Type":"ContainerDied","Data":"381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9"}
Oct 30 18:06:56.289824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.289723    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz" event={"ID":"b0d64771-f9aa-462c-bd01-2fa4d430d7df","Type":"ContainerDied","Data":"8b15f9996493399845f98f3b6cc473709532f08ed39c774d7f3f8cd860995b27"}
Oct 30 18:06:56.289824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.289758    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:06:56.289824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.289800    5534 scope.go:122] "RemoveContainer" containerID="381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9"
Oct 30 18:06:56.292551 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.292270    5534 generic.go:356] "Generic (PLEG): container finished" podID="82557d68-0eee-480b-960b-90087e394d52" containerID="b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263" exitCode=0
Oct 30 18:06:56.292757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.292735    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg" event={"ID":"82557d68-0eee-480b-960b-90087e394d52","Type":"ContainerDied","Data":"b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263"}
Oct 30 18:06:56.292834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.292787    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg" event={"ID":"82557d68-0eee-480b-960b-90087e394d52","Type":"ContainerDied","Data":"40c6877daa9fc90dc5905e78118d25dee2431cdc9686baa4befb2aa43fad6492"}
Oct 30 18:06:56.293096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.292919    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"
Oct 30 18:06:56.295361 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.295341    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr\" (UniqueName: \"kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr\") pod \"82557d68-0eee-480b-960b-90087e394d52\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") "
Oct 30 18:06:56.295486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.295459    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume\") pod \"82557d68-0eee-480b-960b-90087e394d52\" (UID: \"82557d68-0eee-480b-960b-90087e394d52\") "
Oct 30 18:06:56.296030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.295690    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d64771-f9aa-462c-bd01-2fa4d430d7df-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:56.296030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.295705    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpgwv\" (UniqueName: \"kubernetes.io/projected/b0d64771-f9aa-462c-bd01-2fa4d430d7df-kube-api-access-dpgwv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:56.296030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.295979    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume" pod "82557d68-0eee-480b-960b-90087e394d52" (UID: "82557d68-0eee-480b-960b-90087e394d52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:06:56.297575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.297553    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr" pod "82557d68-0eee-480b-960b-90087e394d52" (UID: "82557d68-0eee-480b-960b-90087e394d52"). InnerVolumeSpecName "kube-api-access-qvvzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:06:56.316504 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.316486    5534 scope.go:122] "RemoveContainer" containerID="381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9"
Oct 30 18:06:56.316939 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:56.316898    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9\": container with ID starting with 381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9 not found: ID does not exist" containerID="381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9"
Oct 30 18:06:56.317048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.316950    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9"} err="failed to get container status \"381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9\": rpc error: code = NotFound desc = could not find container \"381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9\": container with ID starting with 381adfa9d0275108a0494f0d2417a73240f809cc8298822dd33dc5cdadfc83c9 not found: ID does not exist"
Oct 30 18:06:56.317048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.316974    5534 scope.go:122] "RemoveContainer" containerID="b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263"
Oct 30 18:06:56.330117 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.330034    5534 scope.go:122] "RemoveContainer" containerID="b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263"
Oct 30 18:06:56.330384 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:06:56.330322    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263\": container with ID starting with b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263 not found: ID does not exist" containerID="b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263"
Oct 30 18:06:56.330384 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.330352    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263"} err="failed to get container status \"b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263\": rpc error: code = NotFound desc = could not find container \"b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263\": container with ID starting with b825107285d0eda51f6f2d05a7cff0a652fe6adc32464dff4dba89737f198263 not found: ID does not exist"
Oct 30 18:06:56.391583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.391507    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"]
Oct 30 18:06:56.391583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.391548    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8d946d-5c76fb5db4-vnvmz"]
Oct 30 18:06:56.396897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.396867    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvzr\" (UniqueName: \"kubernetes.io/projected/82557d68-0eee-480b-960b-90087e394d52-kube-api-access-qvvzr\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:56.396897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.396896    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82557d68-0eee-480b-960b-90087e394d52-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:06:56.419186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.419162    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body=
Oct 30 18:06:56.419304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.419217    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused"
Oct 30 18:06:56.632684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.632628    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"]
Oct 30 18:06:56.636838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:56.636814    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8d946d-64964dbd95-d9lqg"]
Oct 30 18:06:57.419355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:57.419322    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body=
Oct 30 18:06:57.419572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:57.419382    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused"
Oct 30 18:06:57.779244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:57.779133    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:06:58.388093 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:58.388059    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82557d68-0eee-480b-960b-90087e394d52" path="/var/lib/kubelet/pods/82557d68-0eee-480b-960b-90087e394d52/volumes"
Oct 30 18:06:58.388565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:58.388548    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d64771-f9aa-462c-bd01-2fa4d430d7df" path="/var/lib/kubelet/pods/b0d64771-f9aa-462c-bd01-2fa4d430d7df/volumes"
Oct 30 18:06:58.420011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:58.419976    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body=
Oct 30 18:06:58.420174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:58.420036    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused"
Oct 30 18:06:59.419182 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:59.419142    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body=
Oct 30 18:06:59.419744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:59.419215    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused"
Oct 30 18:06:59.820068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:06:59.819956    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:00.194952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.194911    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" containerID="cri-o://bca7bf24103a2c7f91f1f84336205d6f9652b834723c4f796c075b43035d8c3d" gracePeriod=45
Oct 30 18:07:00.328161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.328119    5534 generic.go:356] "Generic (PLEG): container finished" podID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerID="bca7bf24103a2c7f91f1f84336205d6f9652b834723c4f796c075b43035d8c3d" exitCode=0
Oct 30 18:07:00.328315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.328187    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" event={"ID":"46fcbe49-cbaf-4844-9866-efb0a540e624","Type":"ContainerDied","Data":"bca7bf24103a2c7f91f1f84336205d6f9652b834723c4f796c075b43035d8c3d"}
Oct 30 18:07:00.424236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.424202    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:07:00.425889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.425825    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:07:00.463775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.463755    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 18:07:00.551473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.550945    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home\") pod \"46fcbe49-cbaf-4844-9866-efb0a540e624\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") "
Oct 30 18:07:00.551473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.550993    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm\") pod \"46fcbe49-cbaf-4844-9866-efb0a540e624\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") "
Oct 30 18:07:00.551473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.551052    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache\") pod \"46fcbe49-cbaf-4844-9866-efb0a540e624\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") "
Oct 30 18:07:00.551473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.551210    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir\") pod \"46fcbe49-cbaf-4844-9866-efb0a540e624\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") "
Oct 30 18:07:00.551473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.551258    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs\") pod \"46fcbe49-cbaf-4844-9866-efb0a540e624\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") "
Oct 30 18:07:00.551473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.551288    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv\" (UniqueName: \"kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv\") pod \"46fcbe49-cbaf-4844-9866-efb0a540e624\" (UID: \"46fcbe49-cbaf-4844-9866-efb0a540e624\") "
Oct 30 18:07:00.552050 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.551515    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home" pod "46fcbe49-cbaf-4844-9866-efb0a540e624" (UID: "46fcbe49-cbaf-4844-9866-efb0a540e624"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:07:00.552050 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.551886    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:00.552363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.552255    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir" pod "46fcbe49-cbaf-4844-9866-efb0a540e624" (UID: "46fcbe49-cbaf-4844-9866-efb0a540e624"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:07:00.552363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.552322    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache" pod "46fcbe49-cbaf-4844-9866-efb0a540e624" (UID: "46fcbe49-cbaf-4844-9866-efb0a540e624"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:07:00.555556 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.555508    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs" pod "46fcbe49-cbaf-4844-9866-efb0a540e624" (UID: "46fcbe49-cbaf-4844-9866-efb0a540e624"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:00.570271 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.568053    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm" pod "46fcbe49-cbaf-4844-9866-efb0a540e624" (UID: "46fcbe49-cbaf-4844-9866-efb0a540e624"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:07:00.570271 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.568152    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv" pod "46fcbe49-cbaf-4844-9866-efb0a540e624" (UID: "46fcbe49-cbaf-4844-9866-efb0a540e624"). InnerVolumeSpecName "kube-api-access-p7rzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:07:00.652720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.652689    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:00.652720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.652721    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcbe49-cbaf-4844-9866-efb0a540e624-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:00.652934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.652733    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rzv\" (UniqueName: \"kubernetes.io/projected/46fcbe49-cbaf-4844-9866-efb0a540e624-kube-api-access-p7rzv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:00.652934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.652746    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:00.652934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.652760    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcbe49-cbaf-4844-9866-efb0a540e624-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:00.797295 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.797189    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:00.912183 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:00.912106    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:01.148393 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.148367    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:01.336840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.336808    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"
Oct 30 18:07:01.337052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.336839    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" event={"ID":"46fcbe49-cbaf-4844-9866-efb0a540e624","Type":"ContainerDied","Data":"991b33f7b1f15da33d66deedbe687cab108b05e32825ecc13ec1fedd23284b86"}
Oct 30 18:07:01.337052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.336897    5534 scope.go:122] "RemoveContainer" containerID="bca7bf24103a2c7f91f1f84336205d6f9652b834723c4f796c075b43035d8c3d"
Oct 30 18:07:01.388856 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.388826    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"]
Oct 30 18:07:01.400684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.400622    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks"]
Oct 30 18:07:01.448695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:01.448655    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/premium-w6-8d946d-kserve-5c79445f8f-49sks" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:07:02.370633 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:02.370588    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" path="/var/lib/kubelet/pods/46fcbe49-cbaf-4844-9866-efb0a540e624/volumes"
Oct 30 18:07:02.631106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:02.630996    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:02.744063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:02.744028    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:02.885510 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:02.885384    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:03.135846 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.135703    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"]
Oct 30 18:07:03.136934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.136906    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b0d64771-f9aa-462c-bd01-2fa4d430d7df" containerName="payload-processing"
Oct 30 18:07:03.137150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137124    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d64771-f9aa-462c-bd01-2fa4d430d7df" containerName="payload-processing"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137195    5534 cpu_manager.go:414] "containerMap: removing container" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137203    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137227    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9b072561-7419-4119-a650-202284e75d91" containerName="revoke-keys"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137235    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b072561-7419-4119-a650-202284e75d91" containerName="revoke-keys"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137259    5534 cpu_manager.go:414] "containerMap: removing container" podUID="82557d68-0eee-480b-960b-90087e394d52" containerName="payload-pre-processing"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137267    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82557d68-0eee-480b-960b-90087e394d52" containerName="payload-pre-processing"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137291    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2562b78e-3f04-4820-9bbc-7d0399849c16" containerName="revoke-keys"
Oct 30 18:07:03.137297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137299    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2562b78e-3f04-4820-9bbc-7d0399849c16" containerName="revoke-keys"
Oct 30 18:07:03.137832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137568    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d64771-f9aa-462c-bd01-2fa4d430d7df" containerName="payload-processing"
Oct 30 18:07:03.137832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137602    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fcbe49-cbaf-4844-9866-efb0a540e624" containerName="main"
Oct 30 18:07:03.137832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137613    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b072561-7419-4119-a650-202284e75d91" containerName="revoke-keys"
Oct 30 18:07:03.137832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137631    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="82557d68-0eee-480b-960b-90087e394d52" containerName="payload-pre-processing"
Oct 30 18:07:03.137832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.137666    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2562b78e-3f04-4820-9bbc-7d0399849c16" containerName="revoke-keys"
Oct 30 18:07:03.144565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.144537    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.159708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.159680    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"]
Oct 30 18:07:03.165471 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.165433    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d323f5-dockercfg-9sm4n\""
Oct 30 18:07:03.166407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.166073    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d323f5\""
Oct 30 18:07:03.185031 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.184983    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzss\" (UniqueName: \"kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.185232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.185206    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.286677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.286605    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.286884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.286793    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzss\" (UniqueName: \"kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.287384 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.287358    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.327165 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.327130    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzss\" (UniqueName: \"kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss\") pod \"payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.330629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.330597    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"]
Oct 30 18:07:03.334779 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.334760    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.357541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.357076    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"]
Oct 30 18:07:03.388578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.388467    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqqz\" (UniqueName: \"kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz\") pod \"payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.388578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.388539    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.460852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.460810    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:03.489539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.489498    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqqz\" (UniqueName: \"kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz\") pod \"payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.489754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.489564    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.491173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.490557    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.516232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.516201    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqqz\" (UniqueName: \"kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz\") pod \"payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.535318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.534088    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"]
Oct 30 18:07:03.547795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.543679    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.564577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.562784    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:07:03.569636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.566774    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"]
Oct 30 18:07:03.591583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.591508    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.591583 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.591592    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.591907 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.591732    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l98m\" (UniqueName: \"kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.650810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.650713    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:03.660107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.660072    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:07:03.692909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.692877    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.693089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.692935    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.693089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.693034    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l98m\" (UniqueName: \"kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.693556 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.693531    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.695586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.695563    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.719766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.719606    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l98m\" (UniqueName: \"kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m\") pod \"maas-api-revoke-sub-3f00c798-shsmp\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.779986 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.779947    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"]
Oct 30 18:07:03.787406 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:07:03.787381    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74a7a13_47cc_467d_8f6f_7e6f1242bc3f.slice/crio-9371f53a5a206577061c51240118f411ee5b4436d3ea6cffa2c88d7632d8d76c WatchSource:0}: Error finding container 9371f53a5a206577061c51240118f411ee5b4436d3ea6cffa2c88d7632d8d76c: Status 404 returned error can't find the container with id 9371f53a5a206577061c51240118f411ee5b4436d3ea6cffa2c88d7632d8d76c
Oct 30 18:07:03.869475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.869440    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:03.902696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.902592    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:03.992489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:03.992200    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"]
Oct 30 18:07:04.109385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.109356    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:04.179898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.179615    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"]
Oct 30 18:07:04.372226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.372192    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" event={"ID":"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f","Type":"ContainerStarted","Data":"9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa"}
Oct 30 18:07:04.372724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.372698    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:04.372854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.372740    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" event={"ID":"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f","Type":"ContainerStarted","Data":"9371f53a5a206577061c51240118f411ee5b4436d3ea6cffa2c88d7632d8d76c"}
Oct 30 18:07:04.372854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.372765    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp" event={"ID":"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b","Type":"ContainerStarted","Data":"9b32a4cd8af4bd9ccec7fce9d4cee89447c7821068c8e38ca695feaae60064e9"}
Oct 30 18:07:04.374233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.374209    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2" event={"ID":"096b55d1-c745-4df0-9bd5-ca355778385d","Type":"ContainerStarted","Data":"a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7"}
Oct 30 18:07:04.374383 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.374367    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2" event={"ID":"096b55d1-c745-4df0-9bd5-ca355778385d","Type":"ContainerStarted","Data":"fc4edb0920a2137b0d878142146f8d348ddbbd9ee9c81d15293b8cf327febfdd"}
Oct 30 18:07:04.374478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.374401    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:04.413038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.412991    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" podStartSLOduration=1.41297845 podStartE2EDuration="1.41297845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:04.410765499 +0000 UTC m=+3230.172809314" watchObservedRunningTime="2026-10-30 18:07:04.41297845 +0000 UTC m=+3230.175022262"
Oct 30 18:07:04.454773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.454728    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2" podStartSLOduration=1.454715735 podStartE2EDuration="1.454715735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:04.45152503 +0000 UTC m=+3230.213568844" watchObservedRunningTime="2026-10-30 18:07:04.454715735 +0000 UTC m=+3230.216759547"
Oct 30 18:07:04.606585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.606544    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:07:04.909285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:04.909260    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 18:07:05.021113 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.021025    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume\") pod \"a3ad5c83-e9c0-497f-b612-496219e88040\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") "
Oct 30 18:07:05.021113 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.021096    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7\" (UniqueName: \"kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7\") pod \"a3ad5c83-e9c0-497f-b612-496219e88040\" (UID: \"a3ad5c83-e9c0-497f-b612-496219e88040\") "
Oct 30 18:07:05.021490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.021465    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume" pod "a3ad5c83-e9c0-497f-b612-496219e88040" (UID: "a3ad5c83-e9c0-497f-b612-496219e88040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:07:05.021727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.021702    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ad5c83-e9c0-497f-b612-496219e88040-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:05.023380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.023358    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7" pod "a3ad5c83-e9c0-497f-b612-496219e88040" (UID: "a3ad5c83-e9c0-497f-b612-496219e88040"). InnerVolumeSpecName "kube-api-access-pq2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:07:05.122352 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.122316    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2z7\" (UniqueName: \"kubernetes.io/projected/a3ad5c83-e9c0-497f-b612-496219e88040-kube-api-access-pq2z7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:05.385732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.385696    5534 generic.go:356] "Generic (PLEG): container finished" podID="a3ad5c83-e9c0-497f-b612-496219e88040" containerID="62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00" exitCode=0
Oct 30 18:07:05.385923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.385813    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"
Oct 30 18:07:05.385923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.385841    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" event={"ID":"a3ad5c83-e9c0-497f-b612-496219e88040","Type":"ContainerDied","Data":"62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00"}
Oct 30 18:07:05.385923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.385912    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454" event={"ID":"a3ad5c83-e9c0-497f-b612-496219e88040","Type":"ContainerDied","Data":"67be9eb58c61892073344fd15b5df5cec9d94db469a312ad0e372023ed5e48fb"}
Oct 30 18:07:05.386144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.385942    5534 scope.go:122] "RemoveContainer" containerID="62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00"
Oct 30 18:07:05.388371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.388349    5534 generic.go:356] "Generic (PLEG): container finished" podID="284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" containerID="08aeef3885eb45d1b857521cc44095dd6c1bf66d747570f80e0068bdb684b843" exitCode=0
Oct 30 18:07:05.388603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.388389    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp" event={"ID":"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b","Type":"ContainerDied","Data":"08aeef3885eb45d1b857521cc44095dd6c1bf66d747570f80e0068bdb684b843"}
Oct 30 18:07:05.399048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.399027    5534 scope.go:122] "RemoveContainer" containerID="62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00"
Oct 30 18:07:05.399338 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:07:05.399314    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00\": container with ID starting with 62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00 not found: ID does not exist" containerID="62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00"
Oct 30 18:07:05.399429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.399347    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00"} err="failed to get container status \"62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00\": rpc error: code = NotFound desc = could not find container \"62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00\": container with ID starting with 62fa8cd4fbb599478641acf95728a0d06a0619b736ed15bfbb3e8d839c483d00 not found: ID does not exist"
Oct 30 18:07:05.489138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.484674    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"]
Oct 30 18:07:05.491284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.491252    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-506124-59f48f87f5-29454"]
Oct 30 18:07:05.572207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:05.572172    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:06.371814 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.371781    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" path="/var/lib/kubelet/pods/a3ad5c83-e9c0-497f-b612-496219e88040/volumes"
Oct 30 18:07:06.575983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.575957    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:06.639342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.639241    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca\") pod \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") "
Oct 30 18:07:06.639506 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.639478    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m\" (UniqueName: \"kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m\") pod \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") "
Oct 30 18:07:06.639569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.639554    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca\") pod \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\" (UID: \"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b\") "
Oct 30 18:07:06.640018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.639991    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca" pod "284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" (UID: "284aac15-7f8e-43d9-8daf-4c6ae33e0a1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:07:06.642461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.642437    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca" pod "284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" (UID: "284aac15-7f8e-43d9-8daf-4c6ae33e0a1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:06.642692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.642672    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m" pod "284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" (UID: "284aac15-7f8e-43d9-8daf-4c6ae33e0a1b"). InnerVolumeSpecName "kube-api-access-7l98m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:07:06.741592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.741554    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:06.741592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.741589    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:06.741592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:06.741604    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l98m\" (UniqueName: \"kubernetes.io/projected/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b-kube-api-access-7l98m\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:07.408965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:07.408931    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"
Oct 30 18:07:07.409351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:07.408964    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp" event={"ID":"284aac15-7f8e-43d9-8daf-4c6ae33e0a1b","Type":"ContainerDied","Data":"9b32a4cd8af4bd9ccec7fce9d4cee89447c7821068c8e38ca695feaae60064e9"}
Oct 30 18:07:07.409351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:07.409012    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b32a4cd8af4bd9ccec7fce9d4cee89447c7821068c8e38ca695feaae60064e9"
Oct 30 18:07:07.550552 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:07.550518    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:08.925299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:08.925269    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:09.467842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:09.467799    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:09.610447 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:09.610415    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:09.635304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:09.635271    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"]
Oct 30 18:07:09.639487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:09.639453    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-506124-zh2qx"]
Oct 30 18:07:09.756278 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:09.756195    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:10.187432 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:10.187389    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:10.370896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:10.370861    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1859fe4-820e-44cb-8bd8-82051ecddaca" path="/var/lib/kubelet/pods/c1859fe4-820e-44cb-8bd8-82051ecddaca/volumes"
Oct 30 18:07:10.982549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:10.982515    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:11.099441 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:11.099411    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:11.509956 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:11.509923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:12.494978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:12.494947    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:13.078876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:13.078836    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:14.915562 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:14.915531    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:15.247156 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:07:15.247076    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:07:15.391002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:15.390969    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:07:16.216172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.216130    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"]
Oct 30 18:07:16.216579 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.216354    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt" podUID="50b8209a-7697-4c09-b040-42398290a1d4" containerName="maas-api" containerID="cri-o://0d9a60971c79154b746d173a649095eb70165eda26204175f1f6d68c4dd137b6" gracePeriod=30
Oct 30 18:07:16.284170 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.284140    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:16.401856 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.401730    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:07:16.472068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.471975    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:16.487325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.487297    5534 generic.go:356] "Generic (PLEG): container finished" podID="50b8209a-7697-4c09-b040-42398290a1d4" containerID="0d9a60971c79154b746d173a649095eb70165eda26204175f1f6d68c4dd137b6" exitCode=0
Oct 30 18:07:16.487466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.487370    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt" event={"ID":"50b8209a-7697-4c09-b040-42398290a1d4","Type":"ContainerDied","Data":"0d9a60971c79154b746d173a649095eb70165eda26204175f1f6d68c4dd137b6"}
Oct 30 18:07:16.487466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.487419    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt" event={"ID":"50b8209a-7697-4c09-b040-42398290a1d4","Type":"ContainerDied","Data":"a74374e68f29e29b5d72411aeab2011eb80d59d9ac272dc8c4b41b305eab6340"}
Oct 30 18:07:16.487466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.487436    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a74374e68f29e29b5d72411aeab2011eb80d59d9ac272dc8c4b41b305eab6340"
Oct 30 18:07:16.505606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.505581    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:07:16.584163 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.584136    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:16.670183 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.670147    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls\") pod \"50b8209a-7697-4c09-b040-42398290a1d4\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") "
Oct 30 18:07:16.671727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.671171    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl\" (UniqueName: \"kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl\") pod \"50b8209a-7697-4c09-b040-42398290a1d4\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") "
Oct 30 18:07:16.671897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.671872    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs\") pod \"50b8209a-7697-4c09-b040-42398290a1d4\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") "
Oct 30 18:07:16.672324 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.672299    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca\") pod \"50b8209a-7697-4c09-b040-42398290a1d4\" (UID: \"50b8209a-7697-4c09-b040-42398290a1d4\") "
Oct 30 18:07:16.672833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.672788    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca" pod "50b8209a-7697-4c09-b040-42398290a1d4" (UID: "50b8209a-7697-4c09-b040-42398290a1d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:07:16.675143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.675115    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8209a-7697-4c09-b040-42398290a1d4-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:16.675563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.675520    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs" pod "50b8209a-7697-4c09-b040-42398290a1d4" (UID: "50b8209a-7697-4c09-b040-42398290a1d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:16.705912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.705856    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls" pod "50b8209a-7697-4c09-b040-42398290a1d4" (UID: "50b8209a-7697-4c09-b040-42398290a1d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:16.710558 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.710513    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl" pod "50b8209a-7697-4c09-b040-42398290a1d4" (UID: "50b8209a-7697-4c09-b040-42398290a1d4"). InnerVolumeSpecName "kube-api-access-lb5nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:07:16.776281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.776191    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:16.776281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.776220    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb5nl\" (UniqueName: \"kubernetes.io/projected/50b8209a-7697-4c09-b040-42398290a1d4-kube-api-access-lb5nl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:16.776281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.776233    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8209a-7697-4c09-b040-42398290a1d4-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:16.793427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.793395    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:16.904470 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.904343    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"]
Oct 30 18:07:16.907346 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.907314    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96c514-4sbgc"]
Oct 30 18:07:16.908026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:16.907998    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:17.028039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:17.027940    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:17.319247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:17.319161    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:17.504671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:17.503567    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"
Oct 30 18:07:17.536725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:17.529153    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:17.571181 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:17.571084    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"]
Oct 30 18:07:17.582125 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:17.582092    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-96c514-6866fc6448-tctxt"]
Oct 30 18:07:18.369541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:18.369500    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2562b78e-3f04-4820-9bbc-7d0399849c16" path="/var/lib/kubelet/pods/2562b78e-3f04-4820-9bbc-7d0399849c16/volumes"
Oct 30 18:07:18.370048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:18.370030    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b8209a-7697-4c09-b040-42398290a1d4" path="/var/lib/kubelet/pods/50b8209a-7697-4c09-b040-42398290a1d4/volumes"
Oct 30 18:07:18.573155 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:18.573125    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:18.686849 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:18.686759    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:20.610209 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:20.610182    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:22.277672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:22.277622    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:22.478422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:22.478377    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:22.601875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:22.601844    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:23.464912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:23.464871    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:24.031452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:24.031395    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:24.166247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:24.166215    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:24.348077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:24.347122    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:24.462319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:24.462214    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:25.638954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.638912    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"]
Oct 30 18:07:25.639408 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.639244    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42" podUID="d401a478-02e0-4481-8e59-75a3d5c4caec" containerName="maas-api" containerID="cri-o://b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb" gracePeriod=30
Oct 30 18:07:25.706414 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.706366    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:25.934952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.934923    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 18:07:25.998368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.998317    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca\") pod \"d401a478-02e0-4481-8e59-75a3d5c4caec\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") "
Oct 30 18:07:25.998557 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.998516    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226\" (UniqueName: \"kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226\") pod \"d401a478-02e0-4481-8e59-75a3d5c4caec\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") "
Oct 30 18:07:25.998700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.998569    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls\") pod \"d401a478-02e0-4481-8e59-75a3d5c4caec\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") "
Oct 30 18:07:25.998700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.998618    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs\") pod \"d401a478-02e0-4481-8e59-75a3d5c4caec\" (UID: \"d401a478-02e0-4481-8e59-75a3d5c4caec\") "
Oct 30 18:07:25.999386 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:25.999343    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca" pod "d401a478-02e0-4481-8e59-75a3d5c4caec" (UID: "d401a478-02e0-4481-8e59-75a3d5c4caec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:07:26.002018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.001986    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226" pod "d401a478-02e0-4481-8e59-75a3d5c4caec" (UID: "d401a478-02e0-4481-8e59-75a3d5c4caec"). InnerVolumeSpecName "kube-api-access-ts226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:07:26.002394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.002360    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls" pod "d401a478-02e0-4481-8e59-75a3d5c4caec" (UID: "d401a478-02e0-4481-8e59-75a3d5c4caec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:26.002821 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.002796    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs" pod "d401a478-02e0-4481-8e59-75a3d5c4caec" (UID: "d401a478-02e0-4481-8e59-75a3d5c4caec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:26.099775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.099738    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts226\" (UniqueName: \"kubernetes.io/projected/d401a478-02e0-4481-8e59-75a3d5c4caec-kube-api-access-ts226\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:26.099775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.099774    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:26.099775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.099786    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d401a478-02e0-4481-8e59-75a3d5c4caec-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:26.100081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.099799    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d401a478-02e0-4481-8e59-75a3d5c4caec-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:26.157664 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.157608    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:26.269875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.269791    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:26.276937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.276905    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"]
Oct 30 18:07:26.279288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.279261    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8d946d-29xsf"]
Oct 30 18:07:26.372605 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.372567    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635c72fe-c81e-4b64-8f39-5077c220517c" path="/var/lib/kubelet/pods/635c72fe-c81e-4b64-8f39-5077c220517c/volumes"
Oct 30 18:07:26.394040 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.394010    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:26.577248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.577147    5534 generic.go:356] "Generic (PLEG): container finished" podID="d401a478-02e0-4481-8e59-75a3d5c4caec" containerID="b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb" exitCode=0
Oct 30 18:07:26.577248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.577210    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"
Oct 30 18:07:26.577248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.577235    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42" event={"ID":"d401a478-02e0-4481-8e59-75a3d5c4caec","Type":"ContainerDied","Data":"b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb"}
Oct 30 18:07:26.577563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.577299    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42" event={"ID":"d401a478-02e0-4481-8e59-75a3d5c4caec","Type":"ContainerDied","Data":"5d1cc77a15da59894c9316c438d87374c3e080c90f81f24f3ca0d530171d8d4c"}
Oct 30 18:07:26.577563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.577332    5534 scope.go:122] "RemoveContainer" containerID="b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb"
Oct 30 18:07:26.589959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.589938    5534 scope.go:122] "RemoveContainer" containerID="b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb"
Oct 30 18:07:26.590319 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:07:26.590299    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb\": container with ID starting with b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb not found: ID does not exist" containerID="b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb"
Oct 30 18:07:26.590385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.590329    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb"} err="failed to get container status \"b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb\": rpc error: code = NotFound desc = could not find container \"b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb\": container with ID starting with b012f1ecb780e003584461f49a928d04b890eed53ad303813ea8ef1771d50dbb not found: ID does not exist"
Oct 30 18:07:26.629129 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.629097    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"]
Oct 30 18:07:26.640979 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.640946    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8d946d-7847dfffc7-qkw42"]
Oct 30 18:07:26.767161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:26.767128    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:27.996473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:27.996442    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:28.134350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:28.134314    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:28.370750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:28.370713    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d401a478-02e0-4481-8e59-75a3d5c4caec" path="/var/lib/kubelet/pods/d401a478-02e0-4481-8e59-75a3d5c4caec/volumes"
Oct 30 18:07:29.703128 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:29.703079    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:29.813885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:29.813855    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:31.200133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:31.200092    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:32.717358 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:32.717316    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:32.830783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:32.830750    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:33.070234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:33.070164    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:34.515982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.515953    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:34.865430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.865400    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"]
Oct 30 18:07:34.865957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.865942    5534 cpu_manager.go:414] "containerMap: removing container" podUID="50b8209a-7697-4c09-b040-42398290a1d4" containerName="maas-api"
Oct 30 18:07:34.866018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.865958    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b8209a-7697-4c09-b040-42398290a1d4" containerName="maas-api"
Oct 30 18:07:34.866018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.865982    5534 cpu_manager.go:414] "containerMap: removing container" podUID="284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" containerName="revoke-keys"
Oct 30 18:07:34.866018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.865987    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" containerName="revoke-keys"
Oct 30 18:07:34.866018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866000    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" containerName="payload-pre-processing"
Oct 30 18:07:34.866018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866005    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" containerName="payload-pre-processing"
Oct 30 18:07:34.866018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866019    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d401a478-02e0-4481-8e59-75a3d5c4caec" containerName="maas-api"
Oct 30 18:07:34.866235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866025    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d401a478-02e0-4481-8e59-75a3d5c4caec" containerName="maas-api"
Oct 30 18:07:34.866235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866135    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b8209a-7697-4c09-b040-42398290a1d4" containerName="maas-api"
Oct 30 18:07:34.866235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866149    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" containerName="revoke-keys"
Oct 30 18:07:34.866235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866157    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ad5c83-e9c0-497f-b612-496219e88040" containerName="payload-pre-processing"
Oct 30 18:07:34.866235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.866164    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d401a478-02e0-4481-8e59-75a3d5c4caec" containerName="maas-api"
Oct 30 18:07:34.870994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.870968    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:34.890350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.890325    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"]
Oct 30 18:07:34.898817 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.898787    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:07:34.898967 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:34.898910    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:07:35.012515 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.012478    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.012744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.012722    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6w6d\" (UniqueName: \"kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.012819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.012801    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.113985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.113942    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6w6d\" (UniqueName: \"kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.114189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.114021    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.114189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.114047    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.114928 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.114897    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.116979 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.116887    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.137037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.137010    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6w6d\" (UniqueName: \"kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.185066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.185031    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:35.302287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.302248    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:35.536489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.536463    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"]
Oct 30 18:07:35.662350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:35.662305    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j" event={"ID":"074eb617-bf15-41d3-9e8e-6ffb08dd9c52","Type":"ContainerStarted","Data":"825dea6472d2022d1c1bd4c88023532cde6a5d2c5a1f3554e3f8ff273602eddf"}
Oct 30 18:07:36.673496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:36.673457    5534 generic.go:356] "Generic (PLEG): container finished" podID="074eb617-bf15-41d3-9e8e-6ffb08dd9c52" containerID="eaa6dea8729c41d5736c3bfad1bacb20bfd44bc22fe1f45e9c527add417da0d0" exitCode=0
Oct 30 18:07:36.673980 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:36.673515    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j" event={"ID":"074eb617-bf15-41d3-9e8e-6ffb08dd9c52","Type":"ContainerDied","Data":"eaa6dea8729c41d5736c3bfad1bacb20bfd44bc22fe1f45e9c527add417da0d0"}
Oct 30 18:07:36.824883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:36.824798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:37.846052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.846024    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:37.950332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.950240    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca\") pod \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") "
Oct 30 18:07:37.950512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.950373    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca\") pod \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") "
Oct 30 18:07:37.950512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.950393    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d\" (UniqueName: \"kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d\") pod \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\" (UID: \"074eb617-bf15-41d3-9e8e-6ffb08dd9c52\") "
Oct 30 18:07:37.950740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.950714    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca" pod "074eb617-bf15-41d3-9e8e-6ffb08dd9c52" (UID: "074eb617-bf15-41d3-9e8e-6ffb08dd9c52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:07:37.950843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.950827    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:37.952860 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.952839    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d" pod "074eb617-bf15-41d3-9e8e-6ffb08dd9c52" (UID: "074eb617-bf15-41d3-9e8e-6ffb08dd9c52"). InnerVolumeSpecName "kube-api-access-d6w6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:07:37.952973 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:37.952861    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca" pod "074eb617-bf15-41d3-9e8e-6ffb08dd9c52" (UID: "074eb617-bf15-41d3-9e8e-6ffb08dd9c52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:07:38.051697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:38.051668    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:38.051697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:38.051696    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6w6d\" (UniqueName: \"kubernetes.io/projected/074eb617-bf15-41d3-9e8e-6ffb08dd9c52-kube-api-access-d6w6d\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:07:38.399003 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:38.398967    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:38.691536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:38.691434    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"
Oct 30 18:07:38.691536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:38.691463    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j" event={"ID":"074eb617-bf15-41d3-9e8e-6ffb08dd9c52","Type":"ContainerDied","Data":"825dea6472d2022d1c1bd4c88023532cde6a5d2c5a1f3554e3f8ff273602eddf"}
Oct 30 18:07:38.691536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:38.691515    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="825dea6472d2022d1c1bd4c88023532cde6a5d2c5a1f3554e3f8ff273602eddf"
Oct 30 18:07:40.457537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:40.457501    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:41.937491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:41.937451    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:43.495531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:43.495495    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:45.156808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:45.156767    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:45.246501 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:07:45.246466    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:07:45.442991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:45.442912    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:45.562231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:45.562141    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:46.781892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:46.781854    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:48.032195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:48.032087    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"]
Oct 30 18:07:48.043178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:48.043151    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781bc6ac-jw886"]
Oct 30 18:07:48.348954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:48.348923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:48.368485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:48.368452    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c03e3fa-0223-4985-8d0e-898e13445b72" path="/var/lib/kubelet/pods/8c03e3fa-0223-4985-8d0e-898e13445b72/volumes"
Oct 30 18:07:49.937341 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:49.937307    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:51.775917 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:51.775882    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:52.239879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.239842    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:52.350156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.350122    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"]
Oct 30 18:07:52.356362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.356325    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:52.358976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.358952    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c31c73-dxk5j"]
Oct 30 18:07:52.369530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.369504    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074eb617-bf15-41d3-9e8e-6ffb08dd9c52" path="/var/lib/kubelet/pods/074eb617-bf15-41d3-9e8e-6ffb08dd9c52/volumes"
Oct 30 18:07:52.473696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.473664    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:52.932266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:52.932225    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:54.384370 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:54.384335    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:54.687306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:54.687218    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:56.293833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:56.293797    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:57.855112 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:57.855076    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:58.031896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:58.031857    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"]
Oct 30 18:07:58.044568 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:58.044537    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfafb6be-mpz5x"]
Oct 30 18:07:58.370414 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:58.370380    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec" path="/var/lib/kubelet/pods/8b68ca58-f4f0-4cc5-9d26-9cff202ef0ec/volumes"
Oct 30 18:07:59.275448 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:59.275416    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:59.413254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:59.413216    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:59.553019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:59.552935    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:07:59.682690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:07:59.679455    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:00.110046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.110007    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"]
Oct 30 18:08:00.110680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.110660    5534 cpu_manager.go:414] "containerMap: removing container" podUID="074eb617-bf15-41d3-9e8e-6ffb08dd9c52" containerName="revoke-keys"
Oct 30 18:08:00.110733 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.110681    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074eb617-bf15-41d3-9e8e-6ffb08dd9c52" containerName="revoke-keys"
Oct 30 18:08:00.110807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.110798    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="074eb617-bf15-41d3-9e8e-6ffb08dd9c52" containerName="revoke-keys"
Oct 30 18:08:00.115712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.115693    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.133204 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.133182    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"]
Oct 30 18:08:00.212493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.212458    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:08:00.212775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.212757    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:08:00.244492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.244459    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.244709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.244530    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlmd\" (UniqueName: \"kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.244709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.244690    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.345833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.345793    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.346272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.345885    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlmd\" (UniqueName: \"kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.346272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.345912    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.346729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.346705    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.348692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.348636    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.369001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.368941    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlmd\" (UniqueName: \"kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.429219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.429190    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:00.736868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.736840    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"]
Oct 30 18:08:00.738794 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:08:00.738767    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e11e06_66ed_46b1_9a8a_642a63d714e6.slice/crio-cdb7348b6cdcd4cf281764610a04f687b0f5ee6125ecd76065e5672a6813902c WatchSource:0}: Error finding container cdb7348b6cdcd4cf281764610a04f687b0f5ee6125ecd76065e5672a6813902c: Status 404 returned error can't find the container with id cdb7348b6cdcd4cf281764610a04f687b0f5ee6125ecd76065e5672a6813902c
Oct 30 18:08:00.858273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:00.858237    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc" event={"ID":"22e11e06-66ed-46b1-9a8a-642a63d714e6","Type":"ContainerStarted","Data":"cdb7348b6cdcd4cf281764610a04f687b0f5ee6125ecd76065e5672a6813902c"}
Oct 30 18:08:01.282270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:01.282237    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:01.868016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:01.867927    5534 generic.go:356] "Generic (PLEG): container finished" podID="22e11e06-66ed-46b1-9a8a-642a63d714e6" containerID="8982bacfa2cbd2f800a2b388ce5e53ba54c398aceecdf05bdd90f506d38a5544" exitCode=0
Oct 30 18:08:01.868401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:01.868015    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc" event={"ID":"22e11e06-66ed-46b1-9a8a-642a63d714e6","Type":"ContainerDied","Data":"8982bacfa2cbd2f800a2b388ce5e53ba54c398aceecdf05bdd90f506d38a5544"}
Oct 30 18:08:02.951354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:02.951319    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:03.036285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.036259    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:03.075231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.075011    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca\") pod \"22e11e06-66ed-46b1-9a8a-642a63d714e6\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") "
Oct 30 18:08:03.075231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.075200    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd\" (UniqueName: \"kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd\") pod \"22e11e06-66ed-46b1-9a8a-642a63d714e6\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") "
Oct 30 18:08:03.075511 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.075268    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca\") pod \"22e11e06-66ed-46b1-9a8a-642a63d714e6\" (UID: \"22e11e06-66ed-46b1-9a8a-642a63d714e6\") "
Oct 30 18:08:03.075511 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.075494    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca" pod "22e11e06-66ed-46b1-9a8a-642a63d714e6" (UID: "22e11e06-66ed-46b1-9a8a-642a63d714e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:03.076130 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.075983    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:03.078728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.078701    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca" pod "22e11e06-66ed-46b1-9a8a-642a63d714e6" (UID: "22e11e06-66ed-46b1-9a8a-642a63d714e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:08:03.081060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.081037    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd" pod "22e11e06-66ed-46b1-9a8a-642a63d714e6" (UID: "22e11e06-66ed-46b1-9a8a-642a63d714e6"). InnerVolumeSpecName "kube-api-access-7hlmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:03.177153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.177067    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlmd\" (UniqueName: \"kubernetes.io/projected/22e11e06-66ed-46b1-9a8a-642a63d714e6-kube-api-access-7hlmd\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:03.177153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.177096    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e11e06-66ed-46b1-9a8a-642a63d714e6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:03.885704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.885670    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"
Oct 30 18:08:03.885704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.885689    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc" event={"ID":"22e11e06-66ed-46b1-9a8a-642a63d714e6","Type":"ContainerDied","Data":"cdb7348b6cdcd4cf281764610a04f687b0f5ee6125ecd76065e5672a6813902c"}
Oct 30 18:08:03.886021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:03.885780    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb7348b6cdcd4cf281764610a04f687b0f5ee6125ecd76065e5672a6813902c"
Oct 30 18:08:04.234965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:04.234885    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:04.578876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:04.578788    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:06.610095 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:06.610063    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:08.666271 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:08.666241    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:10.218508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:10.218464    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:10.557724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:10.557601    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"]
Oct 30 18:08:10.589234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:10.589210    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"]
Oct 30 18:08:10.697306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:10.697273    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:10.822452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:10.822366    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:11.779834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:11.779798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:13.084051 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:13.084020    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:13.680244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:13.680212    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:13.791372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:13.791340    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"]
Oct 30 18:08:13.795053 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:13.795029    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:13.803115 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:13.803090    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d53ec1-xlfzc"]
Oct 30 18:08:13.913459 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:13.913426    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:14.308573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:14.308542    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:14.370184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:14.370154    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e11e06-66ed-46b1-9a8a-642a63d714e6" path="/var/lib/kubelet/pods/22e11e06-66ed-46b1-9a8a-642a63d714e6/volumes"
Oct 30 18:08:15.245153 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:15.245112    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:08:15.459424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:15.459397    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:15.560698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:15.560510    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerName="payload-processing" containerID="cri-o://5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4" gracePeriod=25
Oct 30 18:08:15.589459 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:15.589413    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerName="payload-pre-processing" containerID="cri-o://052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55" gracePeriod=25
Oct 30 18:08:16.979566 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:16.979515    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:19.150570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:19.150532    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:20.330446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.330413    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:20.439656 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.439611    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:20.664337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.663181    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"]
Oct 30 18:08:20.664337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.663458    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" containerID="cri-o://5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1" gracePeriod=30
Oct 30 18:08:20.699592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.699559    5534 scope.go:122] "RemoveContainer" containerID="235913b806e3dba04cc1164b22e2f866da6e2b99f5b9939a2e71e4472d13e85e"
Oct 30 18:08:20.710392 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.710368    5534 scope.go:122] "RemoveContainer" containerID="df0651d029d0bf2c7343cbc67512894ffce64cd017917508850c175e75346fc7"
Oct 30 18:08:20.714321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.714295    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:20.721561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:20.721539    5534 scope.go:122] "RemoveContainer" containerID="318ae302e6fef49e500318d69fb324ccc14eeba4e9fa8fa6e97f3515d84b059d"
Oct 30 18:08:21.137628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.137591    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:21.248371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.248322    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:21.359974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.359939    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"]
Oct 30 18:08:21.360708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.360688    5534 cpu_manager.go:414] "containerMap: removing container" podUID="22e11e06-66ed-46b1-9a8a-642a63d714e6" containerName="revoke-keys"
Oct 30 18:08:21.360708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.360710    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e11e06-66ed-46b1-9a8a-642a63d714e6" containerName="revoke-keys"
Oct 30 18:08:21.360894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.360853    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e11e06-66ed-46b1-9a8a-642a63d714e6" containerName="revoke-keys"
Oct 30 18:08:21.365840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.365816    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.388339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.388258    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"]
Oct 30 18:08:21.437558 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.437528    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:08:21.437772 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.437680    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:08:21.505042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.505006    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.505225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.505057    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqlj\" (UniqueName: \"kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.505225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.505197    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.606942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.606904    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.607154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.607037    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.607154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.607067    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqlj\" (UniqueName: \"kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.607623 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.607597    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.609543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.609521    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.630168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.630140    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqlj\" (UniqueName: \"kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj\") pod \"maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:21.683306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.683215    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:22.006206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:21.992966    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"]
Oct 30 18:08:22.036739 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:22.036699    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8" event={"ID":"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4","Type":"ContainerStarted","Data":"8353ba80797787e63510174c06cb66eb66d8ea4b53be26fc33b1708452ecdd7a"}
Oct 30 18:08:22.772509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:22.772470    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:08:22.793880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:22.793846    5534 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:08:22.794055 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:22.793903    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:08:22.895790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:22.895761    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:23.045127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:23.045038    5534 generic.go:356] "Generic (PLEG): container finished" podID="d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" containerID="ac6083522c1c84ed4096bc0b0b8c7f6bccc00c0732c3e13d7475ac14bdcace41" exitCode=0
Oct 30 18:08:23.045308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:23.045127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8" event={"ID":"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4","Type":"ContainerDied","Data":"ac6083522c1c84ed4096bc0b0b8c7f6bccc00c0732c3e13d7475ac14bdcace41"}
Oct 30 18:08:23.966792 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:23.966754    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:08:24.061721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.061695    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc_2f97ec5d-56f2-4d07-b6a8-9c589e434ef0/payload-processing/0.log"
Oct 30 18:08:24.194179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.194153    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:24.338625 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.338528    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj\" (UniqueName: \"kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj\") pod \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") "
Oct 30 18:08:24.338807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.338707    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca\") pod \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") "
Oct 30 18:08:24.338807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.338747    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca\") pod \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\" (UID: \"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4\") "
Oct 30 18:08:24.339158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.339125    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca" pod "d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" (UID: "d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:24.340951 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.340925    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj" pod "d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" (UID: "d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4"). InnerVolumeSpecName "kube-api-access-tkqlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:24.341342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.341316    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca" pod "d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" (UID: "d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:08:24.434178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.434144    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:24.439923 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.439899    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:24.440048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.439925    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:24.440048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.439940    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkqlj\" (UniqueName: \"kubernetes.io/projected/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4-kube-api-access-tkqlj\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:24.742721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.742690    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2_096b55d1-c745-4df0-9bd5-ca355778385d/payload-processing/0.log"
Oct 30 18:08:24.922133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:24.922109    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2_096b55d1-c745-4df0-9bd5-ca355778385d/payload-processing/0.log"
Oct 30 18:08:25.071499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.071415    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"
Oct 30 18:08:25.071499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.071424    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8" event={"ID":"d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4","Type":"ContainerDied","Data":"8353ba80797787e63510174c06cb66eb66d8ea4b53be26fc33b1708452ecdd7a"}
Oct 30 18:08:25.071499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.071494    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8353ba80797787e63510174c06cb66eb66d8ea4b53be26fc33b1708452ecdd7a"
Oct 30 18:08:25.424744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.424714    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"]
Oct 30 18:08:25.425238 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.425224    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" containerName="revoke-keys"
Oct 30 18:08:25.425287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.425240    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" containerName="revoke-keys"
Oct 30 18:08:25.425324 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.425306    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" containerName="revoke-keys"
Oct 30 18:08:25.428907 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.428886    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.445671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.445611    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:08:25.445671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.445613    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:08:25.449110 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.449086    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"]
Oct 30 18:08:25.451366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.451339    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.451483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.451393    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.451483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.451426    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kckv\" (UniqueName: \"kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.552518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.552461    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.552518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.552529    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.552841 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.552555    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kckv\" (UniqueName: \"kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.553382 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.553356    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.555514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.555482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.575871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.575836    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kckv\" (UniqueName: \"kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv\") pod \"maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.742698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.742662    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:25.970500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:25.970468    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 18:08:26.058455 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.058422    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"]
Oct 30 18:08:26.058875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.058852    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.058931    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.058978    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059019    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059083    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059141    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059208    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059294    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.059366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059333    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data\") pod \"b6135862-99d6-40be-af38-b0ee8856ccf7\" (UID: \"b6135862-99d6-40be-af38-b0ee8856ccf7\") "
Oct 30 18:08:26.060472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.059702    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:26.060472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.060173    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:26.060472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.060248    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:26.060932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.060636    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:26.060932 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:08:26.060852    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2699e991_e18f_4bc3_80fe_57283b71253a.slice/crio-46b42f43fcb33f5ce0dc33f99c79da03020e09110d329d5b7dfbf05a30631b34 WatchSource:0}: Error finding container 46b42f43fcb33f5ce0dc33f99c79da03020e09110d329d5b7dfbf05a30631b34: Status 404 returned error can't find the container with id 46b42f43fcb33f5ce0dc33f99c79da03020e09110d329d5b7dfbf05a30631b34
Oct 30 18:08:26.061126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.060932    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:26.062508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.062363    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:26.062508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.062402    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "kube-api-access-lmrld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:26.063363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.063344    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:26.063677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.063621    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo" pod "b6135862-99d6-40be-af38-b0ee8856ccf7" (UID: "b6135862-99d6-40be-af38-b0ee8856ccf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:08:26.085564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.085528    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs" event={"ID":"2699e991-e18f-4bc3-80fe-57283b71253a","Type":"ContainerStarted","Data":"46b42f43fcb33f5ce0dc33f99c79da03020e09110d329d5b7dfbf05a30631b34"}
Oct 30 18:08:26.087686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.087519    5534 generic.go:356] "Generic (PLEG): container finished" podID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerID="5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1" exitCode=0
Oct 30 18:08:26.087686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.087580    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"
Oct 30 18:08:26.087686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.087595    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" event={"ID":"b6135862-99d6-40be-af38-b0ee8856ccf7","Type":"ContainerDied","Data":"5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1"}
Oct 30 18:08:26.087917 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.087748    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn" event={"ID":"b6135862-99d6-40be-af38-b0ee8856ccf7","Type":"ContainerDied","Data":"8409c59df91bcc61e6479fb2d72f3f4828d6d0f0db8670855710959d6805c0b0"}
Oct 30 18:08:26.087917 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.087787    5534 scope.go:122] "RemoveContainer" containerID="5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1"
Oct 30 18:08:26.101220 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.100941    5534 scope.go:122] "RemoveContainer" containerID="5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1"
Oct 30 18:08:26.101376 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:26.101240    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1\": container with ID starting with 5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1 not found: ID does not exist" containerID="5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1"
Oct 30 18:08:26.101376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.101265    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1"} err="failed to get container status \"5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1\": rpc error: code = NotFound desc = could not find container \"5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1\": container with ID starting with 5b7ca8429956637b2f43b65a9947ec90fbaa5b9f30ecdbf022d0527c81c6e1d1 not found: ID does not exist"
Oct 30 18:08:26.106879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.106854    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:26.149279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.149224    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"]
Oct 30 18:08:26.151897 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.151874    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-x2dkn"]
Oct 30 18:08:26.161269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161243    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161269 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161269    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmrld\" (UniqueName: \"kubernetes.io/projected/b6135862-99d6-40be-af38-b0ee8856ccf7-kube-api-access-lmrld\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161280    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161291    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161301    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6135862-99d6-40be-af38-b0ee8856ccf7-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161308    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161317    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161325    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.161486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.161332    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6135862-99d6-40be-af38-b0ee8856ccf7-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:26.307083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.307051    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:26.371900 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:26.371863    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" path="/var/lib/kubelet/pods/b6135862-99d6-40be-af38-b0ee8856ccf7/volumes"
Oct 30 18:08:27.098478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:27.098445    5534 generic.go:356] "Generic (PLEG): container finished" podID="2699e991-e18f-4bc3-80fe-57283b71253a" containerID="43350c2424b173009b7414c7a247e2166020e0cc91e4fbb26d79ac4f0316efc9" exitCode=0
Oct 30 18:08:27.098989 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:27.098536    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs" event={"ID":"2699e991-e18f-4bc3-80fe-57283b71253a","Type":"ContainerDied","Data":"43350c2424b173009b7414c7a247e2166020e0cc91e4fbb26d79ac4f0316efc9"}
Oct 30 18:08:27.840174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:27.840142    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:28.120683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.120631    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:28.249018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.248997    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:28.287133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.287100    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv\" (UniqueName: \"kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv\") pod \"2699e991-e18f-4bc3-80fe-57283b71253a\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") "
Oct 30 18:08:28.287326 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.287144    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca\") pod \"2699e991-e18f-4bc3-80fe-57283b71253a\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") "
Oct 30 18:08:28.287326 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.287200    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca\") pod \"2699e991-e18f-4bc3-80fe-57283b71253a\" (UID: \"2699e991-e18f-4bc3-80fe-57283b71253a\") "
Oct 30 18:08:28.287630 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.287606    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca" pod "2699e991-e18f-4bc3-80fe-57283b71253a" (UID: "2699e991-e18f-4bc3-80fe-57283b71253a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:28.289823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.289795    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca" pod "2699e991-e18f-4bc3-80fe-57283b71253a" (UID: "2699e991-e18f-4bc3-80fe-57283b71253a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:08:28.291318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.291091    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv" pod "2699e991-e18f-4bc3-80fe-57283b71253a" (UID: "2699e991-e18f-4bc3-80fe-57283b71253a"). InnerVolumeSpecName "kube-api-access-9kckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:28.388710 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.388674    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kckv\" (UniqueName: \"kubernetes.io/projected/2699e991-e18f-4bc3-80fe-57283b71253a-kube-api-access-9kckv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:28.388710 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.388706    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:28.388895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:28.388725    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699e991-e18f-4bc3-80fe-57283b71253a-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:29.115990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:29.115949    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs" event={"ID":"2699e991-e18f-4bc3-80fe-57283b71253a","Type":"ContainerDied","Data":"46b42f43fcb33f5ce0dc33f99c79da03020e09110d329d5b7dfbf05a30631b34"}
Oct 30 18:08:29.116188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:29.116006    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"
Oct 30 18:08:29.116188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:29.116014    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46b42f43fcb33f5ce0dc33f99c79da03020e09110d329d5b7dfbf05a30631b34"
Oct 30 18:08:29.382195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:29.382111    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:31.510395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:31.510358    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:31.803479 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:31.803392    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"]
Oct 30 18:08:31.944813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:31.944780    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:32.058334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:32.058246    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:32.772911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:32.772873    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:08:33.652114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:33.652083    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:33.966316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:33.966218    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:08:35.216362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.216325    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:35.869367 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.869335    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 18:08:35.906425 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.906400    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 18:08:35.979332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979302    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume\") pod \"990f7d6f-c384-47ec-a6c8-404f64f18d68\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") "
Oct 30 18:08:35.979539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979383    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume\") pod \"60f780ed-4c3c-4952-a45e-5ea8043574c9\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") "
Oct 30 18:08:35.979539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979427    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts\" (UniqueName: \"kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts\") pod \"60f780ed-4c3c-4952-a45e-5ea8043574c9\" (UID: \"60f780ed-4c3c-4952-a45e-5ea8043574c9\") "
Oct 30 18:08:35.979539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979493    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j\" (UniqueName: \"kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j\") pod \"990f7d6f-c384-47ec-a6c8-404f64f18d68\" (UID: \"990f7d6f-c384-47ec-a6c8-404f64f18d68\") "
Oct 30 18:08:35.979795 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979766    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume" pod "990f7d6f-c384-47ec-a6c8-404f64f18d68" (UID: "990f7d6f-c384-47ec-a6c8-404f64f18d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:35.979921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979887    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume" pod "60f780ed-4c3c-4952-a45e-5ea8043574c9" (UID: "60f780ed-4c3c-4952-a45e-5ea8043574c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:35.979921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.979917    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990f7d6f-c384-47ec-a6c8-404f64f18d68-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:35.981780 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.981755    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts" pod "60f780ed-4c3c-4952-a45e-5ea8043574c9" (UID: "60f780ed-4c3c-4952-a45e-5ea8043574c9"). InnerVolumeSpecName "kube-api-access-zdsts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:35.982251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:35.982221    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j" pod "990f7d6f-c384-47ec-a6c8-404f64f18d68" (UID: "990f7d6f-c384-47ec-a6c8-404f64f18d68"). InnerVolumeSpecName "kube-api-access-npg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:36.081196 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.081110    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f780ed-4c3c-4952-a45e-5ea8043574c9-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:36.081196 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.081140    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdsts\" (UniqueName: \"kubernetes.io/projected/60f780ed-4c3c-4952-a45e-5ea8043574c9-kube-api-access-zdsts\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:36.081196 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.081151    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npg7j\" (UniqueName: \"kubernetes.io/projected/990f7d6f-c384-47ec-a6c8-404f64f18d68-kube-api-access-npg7j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:36.163867 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.163830    5534 generic.go:356] "Generic (PLEG): container finished" podID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerID="5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4" exitCode=0
Oct 30 18:08:36.164052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.163868    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" event={"ID":"990f7d6f-c384-47ec-a6c8-404f64f18d68","Type":"ContainerDied","Data":"5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4"}
Oct 30 18:08:36.164052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.163918    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"
Oct 30 18:08:36.164052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.163963    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc" event={"ID":"990f7d6f-c384-47ec-a6c8-404f64f18d68","Type":"ContainerDied","Data":"cb3ea82fdf70aeb5e7592e018f5c28529df52d73217274a0e3c8ef453b1014f7"}
Oct 30 18:08:36.164052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.163994    5534 scope.go:122] "RemoveContainer" containerID="5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4"
Oct 30 18:08:36.165572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.165547    5534 generic.go:356] "Generic (PLEG): container finished" podID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerID="052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55" exitCode=0
Oct 30 18:08:36.165718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.165593    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"
Oct 30 18:08:36.165813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.165768    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" event={"ID":"60f780ed-4c3c-4952-a45e-5ea8043574c9","Type":"ContainerDied","Data":"052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55"}
Oct 30 18:08:36.165866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.165845    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2" event={"ID":"60f780ed-4c3c-4952-a45e-5ea8043574c9","Type":"ContainerDied","Data":"ce609f63a411a968978dd78898aac5da52b993f2d6768551335375dd22cf9d32"}
Oct 30 18:08:36.175589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.175569    5534 scope.go:122] "RemoveContainer" containerID="5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4"
Oct 30 18:08:36.175928 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:36.175906    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4\": container with ID starting with 5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4 not found: ID does not exist" containerID="5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4"
Oct 30 18:08:36.175991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.175936    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4"} err="failed to get container status \"5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4\": rpc error: code = NotFound desc = could not find container \"5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4\": container with ID starting with 5070fc4febeb7a84ae491ba7fb33a2e1b3606167bad84e66687eb88f8b716cc4 not found: ID does not exist"
Oct 30 18:08:36.175991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.175954    5534 scope.go:122] "RemoveContainer" containerID="052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55"
Oct 30 18:08:36.186833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.186812    5534 scope.go:122] "RemoveContainer" containerID="052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55"
Oct 30 18:08:36.187153 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:36.187133    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55\": container with ID starting with 052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55 not found: ID does not exist" containerID="052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55"
Oct 30 18:08:36.187216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.187161    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55"} err="failed to get container status \"052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55\": rpc error: code = NotFound desc = could not find container \"052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55\": container with ID starting with 052cbc37ebf2a056af242cb7efdebd8a821c82f3e79ea1171ae6b91000953f55 not found: ID does not exist"
Oct 30 18:08:36.213370 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.213335    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"]
Oct 30 18:08:36.217005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.216981    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d53ec1-69d5984955-dpcj2"]
Oct 30 18:08:36.252395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.252364    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"]
Oct 30 18:08:36.256299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.256270    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d53ec1-67cd78656d-6lmxc"]
Oct 30 18:08:36.369465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.369434    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" path="/var/lib/kubelet/pods/60f780ed-4c3c-4952-a45e-5ea8043574c9/volumes"
Oct 30 18:08:36.369832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.369818    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" path="/var/lib/kubelet/pods/990f7d6f-c384-47ec-a6c8-404f64f18d68/volumes"
Oct 30 18:08:36.804117 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.804024    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerName="payload-processing" containerID="cri-o://6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce" gracePeriod=25
Oct 30 18:08:36.891490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:36.891462    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:38.403096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:38.403066    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:39.672248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672219    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"]
Oct 30 18:08:39.672728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672714    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672729    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672742    5534 cpu_manager.go:414] "containerMap: removing container" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerName="payload-pre-processing"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672750    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerName="payload-pre-processing"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672777    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2699e991-e18f-4bc3-80fe-57283b71253a" containerName="revoke-keys"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672782    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2699e991-e18f-4bc3-80fe-57283b71253a" containerName="revoke-keys"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672806    5534 cpu_manager.go:414] "containerMap: removing container" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerName="payload-processing"
Oct 30 18:08:39.672810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672811    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerName="payload-processing"
Oct 30 18:08:39.673086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672882    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2699e991-e18f-4bc3-80fe-57283b71253a" containerName="revoke-keys"
Oct 30 18:08:39.673086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672895    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="990f7d6f-c384-47ec-a6c8-404f64f18d68" containerName="payload-processing"
Oct 30 18:08:39.673086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672904    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6135862-99d6-40be-af38-b0ee8856ccf7" containerName="istio-proxy"
Oct 30 18:08:39.673086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.672909    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f780ed-4c3c-4952-a45e-5ea8043574c9" containerName="payload-pre-processing"
Oct 30 18:08:39.680559 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.680521    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.697171 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.697143    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"]
Oct 30 18:08:39.704947 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.704920    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:08:39.705104 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.704954    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:08:39.822899 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.822866    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.823088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.823066    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.823132 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.823098    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzdv\" (UniqueName: \"kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.924701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.924580    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.924701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.924627    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzdv\" (UniqueName: \"kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.924930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.924714    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.925382 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.925361    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.927224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.927206    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.947888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.947860    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzdv\" (UniqueName: \"kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv\") pod \"maas-api-revoke-sub-2db02afb-9zldm\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:39.998034 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:39.998006    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:40.017809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:40.017784    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:40.305702 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:40.305674    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"]
Oct 30 18:08:40.309464 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:08:40.309439    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96acb75e_3222_4737_97b1_c11821daeda3.slice/crio-d0aeffdbc24c4b63ce3fff5e680e4bfee76f7fd44be31760b2d1d0a6af4b5e64 WatchSource:0}: Error finding container d0aeffdbc24c4b63ce3fff5e680e4bfee76f7fd44be31760b2d1d0a6af4b5e64: Status 404 returned error can't find the container with id d0aeffdbc24c4b63ce3fff5e680e4bfee76f7fd44be31760b2d1d0a6af4b5e64
Oct 30 18:08:41.202603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:41.202564    5534 generic.go:356] "Generic (PLEG): container finished" podID="96acb75e-3222-4737-97b1-c11821daeda3" containerID="bea9db576ff743103d81b72dc63710b32e9bb1439f961d9f3a2f8a05a0eefecf" exitCode=0
Oct 30 18:08:41.202972 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:41.202634    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm" event={"ID":"96acb75e-3222-4737-97b1-c11821daeda3","Type":"ContainerDied","Data":"bea9db576ff743103d81b72dc63710b32e9bb1439f961d9f3a2f8a05a0eefecf"}
Oct 30 18:08:41.202972 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:41.202707    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm" event={"ID":"96acb75e-3222-4737-97b1-c11821daeda3","Type":"ContainerStarted","Data":"d0aeffdbc24c4b63ce3fff5e680e4bfee76f7fd44be31760b2d1d0a6af4b5e64"}
Oct 30 18:08:42.047973 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.047945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:42.344126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.344095    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:42.451149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.451116    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca\") pod \"96acb75e-3222-4737-97b1-c11821daeda3\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") "
Oct 30 18:08:42.451327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.451212    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca\") pod \"96acb75e-3222-4737-97b1-c11821daeda3\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") "
Oct 30 18:08:42.451424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.451361    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv\" (UniqueName: \"kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv\") pod \"96acb75e-3222-4737-97b1-c11821daeda3\" (UID: \"96acb75e-3222-4737-97b1-c11821daeda3\") "
Oct 30 18:08:42.451565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.451543    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca" pod "96acb75e-3222-4737-97b1-c11821daeda3" (UID: "96acb75e-3222-4737-97b1-c11821daeda3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:42.451722 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.451707    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:42.453276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.453254    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca" pod "96acb75e-3222-4737-97b1-c11821daeda3" (UID: "96acb75e-3222-4737-97b1-c11821daeda3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:08:42.453544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.453524    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv" pod "96acb75e-3222-4737-97b1-c11821daeda3" (UID: "96acb75e-3222-4737-97b1-c11821daeda3"). InnerVolumeSpecName "kube-api-access-ptzdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:42.553308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.553279    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptzdv\" (UniqueName: \"kubernetes.io/projected/96acb75e-3222-4737-97b1-c11821daeda3-kube-api-access-ptzdv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:42.553308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.553303    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96acb75e-3222-4737-97b1-c11821daeda3-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:42.594117 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:42.594073    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:08:43.217662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:43.217614    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"
Oct 30 18:08:43.217818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:43.217632    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm" event={"ID":"96acb75e-3222-4737-97b1-c11821daeda3","Type":"ContainerDied","Data":"d0aeffdbc24c4b63ce3fff5e680e4bfee76f7fd44be31760b2d1d0a6af4b5e64"}
Oct 30 18:08:43.217818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:43.217709    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0aeffdbc24c4b63ce3fff5e680e4bfee76f7fd44be31760b2d1d0a6af4b5e64"
Oct 30 18:08:43.598063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:43.598034    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:43.615886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:43.615860    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"]
Oct 30 18:08:43.616142 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:43.616115    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy" containerID="cri-o://ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278" gracePeriod=30
Oct 30 18:08:45.244404 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:45.244365    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:08:45.286794 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:45.286766    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:46.810466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:46.810400    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:48.317734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:48.317703    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:48.870188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:48.870166    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 18:08:49.024417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024338    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024372    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024394    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024415    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024552    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024619    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024693    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.024819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024717    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:49.024819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024758    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.025084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024854    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:49.025084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024866    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:49.025084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024896    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z\") pod \"07bfc8c2-a073-4c16-b984-282c82d720e5\" (UID: \"07bfc8c2-a073-4c16-b984-282c82d720e5\") "
Oct 30 18:08:49.025084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024926    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:49.025084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.024945    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:49.025354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.025267    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bfc8c2-a073-4c16-b984-282c82d720e5-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.025354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.025288    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.025354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.025303    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.025354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.025318    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.025354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.025332    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.026858 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.026831    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:49.027049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.027026    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:08:49.027190 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.027171    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:08:49.027297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.027283    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z" pod "07bfc8c2-a073-4c16-b984-282c82d720e5" (UID: "07bfc8c2-a073-4c16-b984-282c82d720e5"). InnerVolumeSpecName "kube-api-access-kn22z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:49.126696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.126634    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn22z\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-kube-api-access-kn22z\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.126696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.126691    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.126696 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.126704    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.126934 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.126716    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bfc8c2-a073-4c16-b984-282c82d720e5-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:49.258424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.258386    5534 generic.go:356] "Generic (PLEG): container finished" podID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerID="ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278" exitCode=0
Oct 30 18:08:49.258605 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.258437    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"
Oct 30 18:08:49.258605 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.258473    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" event={"ID":"07bfc8c2-a073-4c16-b984-282c82d720e5","Type":"ContainerDied","Data":"ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278"}
Oct 30 18:08:49.258605 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.258533    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx" event={"ID":"07bfc8c2-a073-4c16-b984-282c82d720e5","Type":"ContainerDied","Data":"99e7a7b86dace30f73e2cb36d5090f90abd2bc18d1b21d1c41a848646e17ed6b"}
Oct 30 18:08:49.258605 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.258559    5534 scope.go:122] "RemoveContainer" containerID="ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278"
Oct 30 18:08:49.272799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.272776    5534 scope.go:122] "RemoveContainer" containerID="e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"
Oct 30 18:08:49.282096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.282080    5534 scope.go:122] "RemoveContainer" containerID="ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278"
Oct 30 18:08:49.282365 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:49.282345    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278\": container with ID starting with ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278 not found: ID does not exist" containerID="ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278"
Oct 30 18:08:49.282439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.282372    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278"} err="failed to get container status \"ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278\": rpc error: code = NotFound desc = could not find container \"ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278\": container with ID starting with ae695234f6044da2f7198bfb9aa20dfb4d12c944732d05fac7f21148e1505278 not found: ID does not exist"
Oct 30 18:08:49.282439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.282388    5534 scope.go:122] "RemoveContainer" containerID="e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"
Oct 30 18:08:49.282691 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:49.282672    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3\": container with ID starting with e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3 not found: ID does not exist" containerID="e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"
Oct 30 18:08:49.282765 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.282697    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3"} err="failed to get container status \"e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3\": rpc error: code = NotFound desc = could not find container \"e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3\": container with ID starting with e5ee00e0674457b0f6288d92bc52c60632d163a288b8c161b3e1e92aedfa1af3 not found: ID does not exist"
Oct 30 18:08:49.309536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.309513    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"]
Oct 30 18:08:49.312686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:49.312668    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-7z5kx"]
Oct 30 18:08:50.368318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:50.368278    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" path="/var/lib/kubelet/pods/07bfc8c2-a073-4c16-b984-282c82d720e5/volumes"
Oct 30 18:08:50.450058 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:50.450021    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:51.928478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:51.928441    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:52.593757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:52.593721    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:08:53.272656 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:53.272605    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:53.701326 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:53.701284    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:53.813001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:53.812974    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:53.816786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:53.816756    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"]
Oct 30 18:08:53.828622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:53.828593    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-27567a-ffmr8"]
Oct 30 18:08:54.334657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:54.334603    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:54.369707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:54.369677    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4" path="/var/lib/kubelet/pods/d3bb0ee0-7ab4-4b78-983b-c35328ee2fd4/volumes"
Oct 30 18:08:54.514556 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:54.514480    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:55.263499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.263445    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:55.346020 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.345992    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"]
Oct 30 18:08:55.346447 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.346192    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc" podUID="2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" containerName="payload-processing" containerID="cri-o://d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b" gracePeriod=30
Oct 30 18:08:55.375913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.375889    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"]
Oct 30 18:08:55.376143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.376115    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4" podUID="a660b3fe-af8a-4332-8c3b-32dfcec760f5" containerName="payload-pre-processing" containerID="cri-o://63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119" gracePeriod=30
Oct 30 18:08:55.434554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.434524    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:55.549900 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.549840    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:55.703592 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.703570    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:08:55.708470 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.708445    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:08:55.802311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.802229    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:55.805469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.805445    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume\") pod \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") "
Oct 30 18:08:55.805629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.805526    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume\") pod \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") "
Oct 30 18:08:55.805629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.805555    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t\" (UniqueName: \"kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t\") pod \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\" (UID: \"a660b3fe-af8a-4332-8c3b-32dfcec760f5\") "
Oct 30 18:08:55.805629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.805596    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq\" (UniqueName: \"kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq\") pod \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\" (UID: \"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0\") "
Oct 30 18:08:55.806007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.805983    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume" pod "2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" (UID: "2f97ec5d-56f2-4d07-b6a8-9c589e434ef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:55.806104 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.805980    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume" pod "a660b3fe-af8a-4332-8c3b-32dfcec760f5" (UID: "a660b3fe-af8a-4332-8c3b-32dfcec760f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:55.812785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.812728    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq" pod "2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" (UID: "2f97ec5d-56f2-4d07-b6a8-9c589e434ef0"). InnerVolumeSpecName "kube-api-access-fmpgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:55.812950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.812818    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t" pod "a660b3fe-af8a-4332-8c3b-32dfcec760f5" (UID: "a660b3fe-af8a-4332-8c3b-32dfcec760f5"). InnerVolumeSpecName "kube-api-access-pt54t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:55.907548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.907517    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpgq\" (UniqueName: \"kubernetes.io/projected/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-kube-api-access-fmpgq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:55.907548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.907546    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a660b3fe-af8a-4332-8c3b-32dfcec760f5-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:55.907548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.907555    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:55.907821 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.907565    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt54t\" (UniqueName: \"kubernetes.io/projected/a660b3fe-af8a-4332-8c3b-32dfcec760f5-kube-api-access-pt54t\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:55.913542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.913509    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:55.926218 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.926188    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"]
Oct 30 18:08:55.931064 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:55.931037    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95f55d-njtvs"]
Oct 30 18:08:56.025695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.025662    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:56.318657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.318598    5534 generic.go:356] "Generic (PLEG): container finished" podID="a660b3fe-af8a-4332-8c3b-32dfcec760f5" containerID="63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119" exitCode=0
Oct 30 18:08:56.318852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.318681    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"
Oct 30 18:08:56.318852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.318685    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4" event={"ID":"a660b3fe-af8a-4332-8c3b-32dfcec760f5","Type":"ContainerDied","Data":"63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119"}
Oct 30 18:08:56.318852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.318742    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4" event={"ID":"a660b3fe-af8a-4332-8c3b-32dfcec760f5","Type":"ContainerDied","Data":"b0a39f3455e5795cef69c59c8cc17310237b2882b01ca0fdc00bb564dc1fef27"}
Oct 30 18:08:56.318852 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.318769    5534 scope.go:122] "RemoveContainer" containerID="63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119"
Oct 30 18:08:56.320160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.320140    5534 generic.go:356] "Generic (PLEG): container finished" podID="2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" containerID="d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b" exitCode=0
Oct 30 18:08:56.320267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.320198    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"
Oct 30 18:08:56.320267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.320203    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc" event={"ID":"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0","Type":"ContainerDied","Data":"d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b"}
Oct 30 18:08:56.320386 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.320276    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc" event={"ID":"2f97ec5d-56f2-4d07-b6a8-9c589e434ef0","Type":"ContainerDied","Data":"06e0c45e4b2f4500d06bd07f2d50319ac028d6c4f53326279c56bc7b32ecd5b9"}
Oct 30 18:08:56.331614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.331590    5534 scope.go:122] "RemoveContainer" containerID="63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119"
Oct 30 18:08:56.332036 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:56.331999    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119\": container with ID starting with 63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119 not found: ID does not exist" containerID="63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119"
Oct 30 18:08:56.332116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.332035    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119"} err="failed to get container status \"63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119\": rpc error: code = NotFound desc = could not find container \"63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119\": container with ID starting with 63b8cf9a027acb696b5b4c2860dfb8faa5fef62b5d4185391cb12b4d8e161119 not found: ID does not exist"
Oct 30 18:08:56.332116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.332060    5534 scope.go:122] "RemoveContainer" containerID="d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b"
Oct 30 18:08:56.341331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.341313    5534 scope.go:122] "RemoveContainer" containerID="d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b"
Oct 30 18:08:56.341626 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:56.341609    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b\": container with ID starting with d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b not found: ID does not exist" containerID="d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b"
Oct 30 18:08:56.341722 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.341635    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b"} err="failed to get container status \"d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b\": rpc error: code = NotFound desc = could not find container \"d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b\": container with ID starting with d301e402848b50d1aafc7c4831e8665646f584c39d8a0e5797b3e49300f79d8b not found: ID does not exist"
Oct 30 18:08:56.370052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.370024    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2699e991-e18f-4bc3-80fe-57283b71253a" path="/var/lib/kubelet/pods/2699e991-e18f-4bc3-80fe-57283b71253a/volumes"
Oct 30 18:08:56.371072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.371051    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"]
Oct 30 18:08:56.383458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.383429    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95f55d-68998c74dc-tfxx4"]
Oct 30 18:08:56.408007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.407975    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"]
Oct 30 18:08:56.411372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.411351    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95f55d-855dfbfc94-6fvtc"]
Oct 30 18:08:56.423021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:56.422992    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:57.030212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.030175    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"]
Oct 30 18:08:57.040538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.040511    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3aa0e1b-g5skj"]
Oct 30 18:08:57.067512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.067450    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 18:08:57.121865 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.121835    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s\" (UniqueName: \"kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s\") pod \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") "
Oct 30 18:08:57.122071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.121951    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume\") pod \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\" (UID: \"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a\") "
Oct 30 18:08:57.122659 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.122619    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume" pod "a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" (UID: "a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:08:57.124889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.124859    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s" pod "a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" (UID: "a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a"). InnerVolumeSpecName "kube-api-access-4dk7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:08:57.223249 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.223206    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dk7s\" (UniqueName: \"kubernetes.io/projected/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-kube-api-access-4dk7s\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:57.223249 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.223239    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:08:57.330309 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.330224    5534 generic.go:356] "Generic (PLEG): container finished" podID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerID="6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce" exitCode=0
Oct 30 18:08:57.330309 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.330272    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"
Oct 30 18:08:57.330525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.330312    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" event={"ID":"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a","Type":"ContainerDied","Data":"6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce"}
Oct 30 18:08:57.330525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.330399    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544" event={"ID":"a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a","Type":"ContainerDied","Data":"1e54da4152bce53ad62e18899620618ccfa5d93386c4709b735ee06f41f1eac6"}
Oct 30 18:08:57.330525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.330425    5534 scope.go:122] "RemoveContainer" containerID="6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce"
Oct 30 18:08:57.341222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.341203    5534 scope.go:122] "RemoveContainer" containerID="6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce"
Oct 30 18:08:57.341480 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:08:57.341458    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce\": container with ID starting with 6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce not found: ID does not exist" containerID="6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce"
Oct 30 18:08:57.341584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.341486    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce"} err="failed to get container status \"6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce\": rpc error: code = NotFound desc = could not find container \"6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce\": container with ID starting with 6f2337373e81bb83ee3075e625ae008af638b7205cf38ff4d43ebacdeec34fce not found: ID does not exist"
Oct 30 18:08:57.380248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.380217    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"]
Oct 30 18:08:57.384395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.384361    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-27567a-59dcdfb666-nn544"]
Oct 30 18:08:57.853966 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:57.853930    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:08:58.368491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:58.368455    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" path="/var/lib/kubelet/pods/2f97ec5d-56f2-4d07-b6a8-9c589e434ef0/volumes"
Oct 30 18:08:58.368938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:58.368924    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59afd1d4-2643-466f-97ff-f6bd1fe39c05" path="/var/lib/kubelet/pods/59afd1d4-2643-466f-97ff-f6bd1fe39c05/volumes"
Oct 30 18:08:58.369266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:58.369254    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" path="/var/lib/kubelet/pods/a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a/volumes"
Oct 30 18:08:58.369715 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:58.369692    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a660b3fe-af8a-4332-8c3b-32dfcec760f5" path="/var/lib/kubelet/pods/a660b3fe-af8a-4332-8c3b-32dfcec760f5/volumes"
Oct 30 18:08:59.887119 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:08:59.887089    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:00.566498 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:00.566459    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:00.685216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:00.685185    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:00.911452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:00.911408    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"]
Oct 30 18:09:00.912330 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:00.912294    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy" containerID="cri-o://222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f" gracePeriod=30
Oct 30 18:09:00.938427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:00.938388    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:01.462368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:01.462338    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:01.601600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:01.601566    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:03.207718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:03.207680    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:03.930602 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:03.930567    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:04.122996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:04.122960    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:04.368633 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:04.368596    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:04.642879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:04.642798    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:04.836433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:04.836401    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:06.191129 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.191051    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 18:09:06.321106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321052    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321185    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321221    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321293    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321321    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321363    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321411    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321419    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:09:06.321694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321441    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321469    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token\") pod \"c688455a-36a2-4f26-9ccd-20be9f3af129\" (UID: \"c688455a-36a2-4f26-9ccd-20be9f3af129\") "
Oct 30 18:09:06.321694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321609    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:09:06.321694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321634    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:09:06.322002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321933    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:09:06.322002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321937    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.322002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.321981    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.322002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.322000    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.322400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.322158    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:09:06.323807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.323771    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:09:06.324071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.324041    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:09:06.324290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.324269    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:09:06.324290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.324276    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc" pod "c688455a-36a2-4f26-9ccd-20be9f3af129" (UID: "c688455a-36a2-4f26-9ccd-20be9f3af129"). InnerVolumeSpecName "kube-api-access-6hnzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:09:06.396886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.396854    5534 generic.go:356] "Generic (PLEG): container finished" podID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerID="222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f" exitCode=0
Oct 30 18:09:06.397065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.396910    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"
Oct 30 18:09:06.397065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.396942    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" event={"ID":"c688455a-36a2-4f26-9ccd-20be9f3af129","Type":"ContainerDied","Data":"222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f"}
Oct 30 18:09:06.397065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.397000    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r" event={"ID":"c688455a-36a2-4f26-9ccd-20be9f3af129","Type":"ContainerDied","Data":"e90a8e01f4ce2d811af80d46298d8fa7b0b3917d15120b4b28233d43ddb51d08"}
Oct 30 18:09:06.397065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.397021    5534 scope.go:122] "RemoveContainer" containerID="222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f"
Oct 30 18:09:06.407566 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.407544    5534 scope.go:122] "RemoveContainer" containerID="a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"
Oct 30 18:09:06.417619 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.417600    5534 scope.go:122] "RemoveContainer" containerID="222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f"
Oct 30 18:09:06.417935 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:06.417907    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f\": container with ID starting with 222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f not found: ID does not exist" containerID="222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f"
Oct 30 18:09:06.418060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.417943    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f"} err="failed to get container status \"222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f\": rpc error: code = NotFound desc = could not find container \"222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f\": container with ID starting with 222d2141e7a140e2d58b083e4a2285930565d0d87f4e00add8cf3c8912bf201f not found: ID does not exist"
Oct 30 18:09:06.418060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.417960    5534 scope.go:122] "RemoveContainer" containerID="a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"
Oct 30 18:09:06.418173 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:06.418157    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5\": container with ID starting with a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5 not found: ID does not exist" containerID="a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"
Oct 30 18:09:06.418215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.418172    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5"} err="failed to get container status \"a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5\": rpc error: code = NotFound desc = could not find container \"a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5\": container with ID starting with a45d0a2b887320b961029bdf238c9238c1f22c2bc2b3e0daeb4a6c787d25dee5 not found: ID does not exist"
Oct 30 18:09:06.422989 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.422968    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c688455a-36a2-4f26-9ccd-20be9f3af129-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.423073 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.422997    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnzc\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-kube-api-access-6hnzc\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.423073 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.423011    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.423073 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.423025    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.423073 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.423037    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.423073 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.423048    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c688455a-36a2-4f26-9ccd-20be9f3af129-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:06.447676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.447581    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"]
Oct 30 18:09:06.454363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.454335    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-kmq9r"]
Oct 30 18:09:06.624107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:06.624073    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:08.041898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:08.041866    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"]
Oct 30 18:09:08.053629 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:08.053604    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa1a826-95srh"]
Oct 30 18:09:08.370688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:08.370629    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb544957-3dfb-4e72-9a0e-22fdecba6088" path="/var/lib/kubelet/pods/bb544957-3dfb-4e72-9a0e-22fdecba6088/volumes"
Oct 30 18:09:08.371127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:08.371109    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" path="/var/lib/kubelet/pods/c688455a-36a2-4f26-9ccd-20be9f3af129/volumes"
Oct 30 18:09:08.559515 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:08.559483    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:10.185596 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:10.185560    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:10.550179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:10.550097    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:11.752274 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:11.752238    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:13.289670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:13.289620    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:15.030598 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:15.030564    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:15.244047 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:15.244012    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:09:15.326566 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:15.326488    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:15.437689 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:15.437625    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:16.964044 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:16.964012    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:18.433957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:18.433927    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:20.004872 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:20.004826    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:20.779886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:20.779852    5534 scope.go:122] "RemoveContainer" containerID="1d0911f407008602c0e71393c4a3384fc1e84243d6680b4653d712ba8d5a4117"
Oct 30 18:09:20.790146 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:20.790119    5534 scope.go:122] "RemoveContainer" containerID="aa58df6e2d5e4d4d744d350b1780a4e876b141f7524fdee7648250a6bf4785b2"
Oct 30 18:09:20.799409 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:20.799388    5534 scope.go:122] "RemoveContainer" containerID="e48af2a68706c29a87aa8b008003a66ad773cf208a938d4808fafa16ecac3060"
Oct 30 18:09:20.808633 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:20.808609    5534 scope.go:122] "RemoveContainer" containerID="2765bac4f7f80fc4ca18836fe653ef95ddc0e22ac051790cfd0df9d4366cd0ff"
Oct 30 18:09:21.494390 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:21.494360    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:23.004003 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:23.003971    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:25.010655 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:25.010594    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:26.460037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:26.459989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:28.095740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:28.095706    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:28.239339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:28.239304    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:29.572277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:29.572237    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:31.300486 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:31.300452    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:33.331434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:33.331397    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:34.839550 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:34.839513    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:36.411505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:36.411476    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:37.933574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:37.933539    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:38.052158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:38.052126    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:39.000069 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:39.000033    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:39.587481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:39.587441    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:39.780403 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:39.780372    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:40.681168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:40.681138    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:41.615540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:41.615500    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:43.090548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:43.090515    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:44.708013 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:44.707980    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:44.867223 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:44.867188    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:44.989087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:44.989012    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:45.237536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:45.237482    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:45.244564 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:45.244491    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:09:46.402143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:46.402103    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:48.029742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:48.029666    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:49.975299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:49.975263    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:51.490059 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:51.490025    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:53.032098 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:53.032062    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:54.598847 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:54.598814    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:55.954559 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:55.954521    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"]
Oct 30 18:09:55.955081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:55.954823    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf" podUID="f3b7764c-3fd0-476f-babd-3ec5b7ba335d" containerName="maas-api" containerID="cri-o://7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca" gracePeriod=30
Oct 30 18:09:56.022790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.022760    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:56.104276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.104210    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"]
Oct 30 18:09:56.104457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.104436    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2" podUID="096b55d1-c745-4df0-9bd5-ca355778385d" containerName="payload-processing" containerID="cri-o://a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7" gracePeriod=30
Oct 30 18:09:56.127360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.126308    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"]
Oct 30 18:09:56.127360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.126568    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" podUID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" containerName="payload-pre-processing" containerID="cri-o://9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa" gracePeriod=30
Oct 30 18:09:56.192522 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.192490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:56.264187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.264157    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:09:56.318627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.318514    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:56.391676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.387714    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:09:56.457342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.457304    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls\") pod \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") "
Oct 30 18:09:56.457531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.457424    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume\") pod \"096b55d1-c745-4df0-9bd5-ca355778385d\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") "
Oct 30 18:09:56.457531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.457481    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl\" (UniqueName: \"kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl\") pod \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") "
Oct 30 18:09:56.457531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.457518    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca\") pod \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") "
Oct 30 18:09:56.457741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.457585    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs\") pod \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\" (UID: \"f3b7764c-3fd0-476f-babd-3ec5b7ba335d\") "
Oct 30 18:09:56.460075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.459533    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume" pod "096b55d1-c745-4df0-9bd5-ca355778385d" (UID: "096b55d1-c745-4df0-9bd5-ca355778385d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:09:56.460075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.459726    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca" pod "f3b7764c-3fd0-476f-babd-3ec5b7ba335d" (UID: "f3b7764c-3fd0-476f-babd-3ec5b7ba335d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:09:56.461886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.461856    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs" pod "f3b7764c-3fd0-476f-babd-3ec5b7ba335d" (UID: "f3b7764c-3fd0-476f-babd-3ec5b7ba335d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:09:56.461886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.461870    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls" pod "f3b7764c-3fd0-476f-babd-3ec5b7ba335d" (UID: "f3b7764c-3fd0-476f-babd-3ec5b7ba335d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:09:56.463786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.463766    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl" pod "f3b7764c-3fd0-476f-babd-3ec5b7ba335d" (UID: "f3b7764c-3fd0-476f-babd-3ec5b7ba335d"). InnerVolumeSpecName "kube-api-access-d6gjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:09:56.467406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.467379    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:09:56.551826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.551750    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:56.559021 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.558999    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz\" (UniqueName: \"kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz\") pod \"096b55d1-c745-4df0-9bd5-ca355778385d\" (UID: \"096b55d1-c745-4df0-9bd5-ca355778385d\") "
Oct 30 18:09:56.559158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559040    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume\") pod \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") "
Oct 30 18:09:56.559158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559138    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss\" (UniqueName: \"kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss\") pod \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\" (UID: \"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f\") "
Oct 30 18:09:56.559472 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559450    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b55d1-c745-4df0-9bd5-ca355778385d-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.559537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559477    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gjl\" (UniqueName: \"kubernetes.io/projected/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-kube-api-access-d6gjl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.559537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559492    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.559537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559506    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.559537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559519    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b7764c-3fd0-476f-babd-3ec5b7ba335d-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.559537 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.559518    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume" pod "d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" (UID: "d74a7a13-47cc-467d-8f6f-7e6f1242bc3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:09:56.561373 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.561348    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz" pod "096b55d1-c745-4df0-9bd5-ca355778385d" (UID: "096b55d1-c745-4df0-9bd5-ca355778385d"). InnerVolumeSpecName "kube-api-access-8vqqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:09:56.561484 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.561386    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss" pod "d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" (UID: "d74a7a13-47cc-467d-8f6f-7e6f1242bc3f"). InnerVolumeSpecName "kube-api-access-mdzss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:09:56.660036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.660012    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzss\" (UniqueName: \"kubernetes.io/projected/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-kube-api-access-mdzss\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.660036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.660036    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqqz\" (UniqueName: \"kubernetes.io/projected/096b55d1-c745-4df0-9bd5-ca355778385d-kube-api-access-8vqqz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.660231 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.660050    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:09:56.664819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.664787    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:56.723248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.723215    5534 generic.go:356] "Generic (PLEG): container finished" podID="f3b7764c-3fd0-476f-babd-3ec5b7ba335d" containerID="7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca" exitCode=0
Oct 30 18:09:56.723439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.723285    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"
Oct 30 18:09:56.723439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.723301    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf" event={"ID":"f3b7764c-3fd0-476f-babd-3ec5b7ba335d","Type":"ContainerDied","Data":"7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca"}
Oct 30 18:09:56.723439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.723366    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf" event={"ID":"f3b7764c-3fd0-476f-babd-3ec5b7ba335d","Type":"ContainerDied","Data":"8b9298998d31cd2e239f1a1406d0004cbc790cfee8c7de9b599daa4f00d2c8f6"}
Oct 30 18:09:56.723439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.723394    5534 scope.go:122] "RemoveContainer" containerID="7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca"
Oct 30 18:09:56.729955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.729888    5534 generic.go:356] "Generic (PLEG): container finished" podID="096b55d1-c745-4df0-9bd5-ca355778385d" containerID="a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7" exitCode=0
Oct 30 18:09:56.730300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.730182    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2" event={"ID":"096b55d1-c745-4df0-9bd5-ca355778385d","Type":"ContainerDied","Data":"a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7"}
Oct 30 18:09:56.730300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.730242    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2" event={"ID":"096b55d1-c745-4df0-9bd5-ca355778385d","Type":"ContainerDied","Data":"fc4edb0920a2137b0d878142146f8d348ddbbd9ee9c81d15293b8cf327febfdd"}
Oct 30 18:09:56.730514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.730341    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"
Oct 30 18:09:56.737127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.737073    5534 generic.go:356] "Generic (PLEG): container finished" podID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" containerID="9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa" exitCode=0
Oct 30 18:09:56.738400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.737176    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" event={"ID":"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f","Type":"ContainerDied","Data":"9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa"}
Oct 30 18:09:56.738400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.737264    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" event={"ID":"d74a7a13-47cc-467d-8f6f-7e6f1242bc3f","Type":"ContainerDied","Data":"9371f53a5a206577061c51240118f411ee5b4436d3ea6cffa2c88d7632d8d76c"}
Oct 30 18:09:56.738400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.737355    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"
Oct 30 18:09:56.741533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.741509    5534 scope.go:122] "RemoveContainer" containerID="7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca"
Oct 30 18:09:56.741903 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:56.741877    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca\": container with ID starting with 7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca not found: ID does not exist" containerID="7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca"
Oct 30 18:09:56.741994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.741918    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca"} err="failed to get container status \"7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca\": rpc error: code = NotFound desc = could not find container \"7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca\": container with ID starting with 7f2d482b7b9288a1e593c367762cf0f93a98e07eecc2e1431c6bef8f5ff2f9ca not found: ID does not exist"
Oct 30 18:09:56.741994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.741943    5534 scope.go:122] "RemoveContainer" containerID="a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7"
Oct 30 18:09:56.758344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.758316    5534 scope.go:122] "RemoveContainer" containerID="a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7"
Oct 30 18:09:56.758773 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:56.758751    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7\": container with ID starting with a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7 not found: ID does not exist" containerID="a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7"
Oct 30 18:09:56.758848 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.758785    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7"} err="failed to get container status \"a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7\": rpc error: code = NotFound desc = could not find container \"a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7\": container with ID starting with a02ccb95b76d837480334fcbb103f4411bdc56b2f439d9abee349c1918149cc7 not found: ID does not exist"
Oct 30 18:09:56.758848 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.758803    5534 scope.go:122] "RemoveContainer" containerID="9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa"
Oct 30 18:09:56.769281 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.769251    5534 scope.go:122] "RemoveContainer" containerID="9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa"
Oct 30 18:09:56.769608 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:09:56.769585    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa\": container with ID starting with 9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa not found: ID does not exist" containerID="9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa"
Oct 30 18:09:56.769734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.769615    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa"} err="failed to get container status \"9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa\": rpc error: code = NotFound desc = could not find container \"9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa\": container with ID starting with 9b2e5cb5d1e863475f1cbd446844a9abbc0565b84209b598f990c0ce0e23d6fa not found: ID does not exist"
Oct 30 18:09:56.779757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.779734    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:56.789963 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.789944    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"]
Oct 30 18:09:56.793464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.793444    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d323f5-7bcf6d5b9-zgqgf"]
Oct 30 18:09:56.840831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.840809    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"]
Oct 30 18:09:56.844894 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.844872    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d323f5-56b77d7ddb-dcxl2"]
Oct 30 18:09:56.877853 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.877830    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"]
Oct 30 18:09:56.884135 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:56.884109    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts"]
Oct 30 18:09:57.399651 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:57.399617    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d323f5-755dc688d7-7ftts" podUID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.1.116:9004: i/o timeout"
Oct 30 18:09:58.210740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:58.210700    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:09:58.369123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:58.369088    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096b55d1-c745-4df0-9bd5-ca355778385d" path="/var/lib/kubelet/pods/096b55d1-c745-4df0-9bd5-ca355778385d/volumes"
Oct 30 18:09:58.369478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:58.369465    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" path="/var/lib/kubelet/pods/d74a7a13-47cc-467d-8f6f-7e6f1242bc3f/volumes"
Oct 30 18:09:58.369863 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:58.369850    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b7764c-3fd0-476f-babd-3ec5b7ba335d" path="/var/lib/kubelet/pods/f3b7764c-3fd0-476f-babd-3ec5b7ba335d/volumes"
Oct 30 18:09:59.705193 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:09:59.705156    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:01.250737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:01.250705    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:02.891733 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:02.891698    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:04.348903 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:04.348869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:04.518035 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:04.517999    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:04.633319 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:04.633225    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:05.625215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:05.625174    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:05.881225 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:05.881141    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:06.485188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:06.485131    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:07.165524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:07.165487    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:07.477983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:07.477901    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:09.091789 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:09.091752    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:10.648019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:10.647975    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:12.211957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:12.211922    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:13.519235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:13.519201    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:13.633172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:13.633135    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:13.822737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:13.813060    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:13.826606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:13.826571    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"]
Oct 30 18:10:13.826955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:13.826905    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" containerID="cri-o://ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a" gracePeriod=30
Oct 30 18:10:13.926853 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:13.926812    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:14.349359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349321    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"]
Oct 30 18:10:14.349775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349760    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349777    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349796    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerName="payload-processing"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349802    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerName="payload-processing"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349810    5534 cpu_manager.go:414] "containerMap: removing container" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349816    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349825    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f3b7764c-3fd0-476f-babd-3ec5b7ba335d" containerName="maas-api"
Oct 30 18:10:14.349832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349830    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b7764c-3fd0-476f-babd-3ec5b7ba335d" containerName="maas-api"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349841    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349846    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349861    5534 cpu_manager.go:414] "containerMap: removing container" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349866    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349874    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a660b3fe-af8a-4332-8c3b-32dfcec760f5" containerName="payload-pre-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349879    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a660b3fe-af8a-4332-8c3b-32dfcec760f5" containerName="payload-pre-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349886    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" containerName="payload-pre-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349890    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" containerName="payload-pre-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349897    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" containerName="payload-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349901    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" containerName="payload-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349910    5534 cpu_manager.go:414] "containerMap: removing container" podUID="96acb75e-3222-4737-97b1-c11821daeda3" containerName="revoke-keys"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349914    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96acb75e-3222-4737-97b1-c11821daeda3" containerName="revoke-keys"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349922    5534 cpu_manager.go:414] "containerMap: removing container" podUID="096b55d1-c745-4df0-9bd5-ca355778385d" containerName="payload-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349927    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096b55d1-c745-4df0-9bd5-ca355778385d" containerName="payload-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349984    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="96acb75e-3222-4737-97b1-c11821daeda3" containerName="revoke-keys"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.349991    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="096b55d1-c745-4df0-9bd5-ca355778385d" containerName="payload-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350000    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350008    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48c1de7-d6c3-4e39-8dc8-507b8ce7f64a" containerName="payload-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350015    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350020    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b7764c-3fd0-476f-babd-3ec5b7ba335d" containerName="maas-api"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350027    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bfc8c2-a073-4c16-b984-282c82d720e5" containerName="istio-proxy"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350032    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a660b3fe-af8a-4332-8c3b-32dfcec760f5" containerName="payload-pre-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350037    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74a7a13-47cc-467d-8f6f-7e6f1242bc3f" containerName="payload-pre-processing"
Oct 30 18:10:14.350065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.350041    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f97ec5d-56f2-4d07-b6a8-9c589e434ef0" containerName="payload-processing"
Oct 30 18:10:14.353409 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.353390    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.374331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.374291    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"]
Oct 30 18:10:14.397273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.397239    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:10:14.402712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.402679    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:10:14.445526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.445488    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvfw\" (UniqueName: \"kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.445760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.445676    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.445842 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.445790    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.547023 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.546984    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.547544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.547067    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvfw\" (UniqueName: \"kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.547544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.547121    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.547757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.547735    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.549721 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.549694    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.570685 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.570601    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvfw\" (UniqueName: \"kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.669922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.669835    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:14.974167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:14.974082    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"]
Oct 30 18:10:14.978431 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:10:14.978400    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a0aff1_9416_41ec_b272_d860b14371d2.slice/crio-fa8f840ea9f119b4392d584123c4f203cbe71f1af8cda4f18086b8fffb6572ca WatchSource:0}: Error finding container fa8f840ea9f119b4392d584123c4f203cbe71f1af8cda4f18086b8fffb6572ca: Status 404 returned error can't find the container with id fa8f840ea9f119b4392d584123c4f203cbe71f1af8cda4f18086b8fffb6572ca
Oct 30 18:10:15.244633 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:15.244549    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:10:15.427729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:15.427690    5534 patch_prober.go:29] interesting pod/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:10:15.427901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:15.427746    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:10:15.532927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:15.532850    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:15.876420 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:15.876383    5534 generic.go:356] "Generic (PLEG): container finished" podID="f6a0aff1-9416-41ec-b272-d860b14371d2" containerID="2806600eb9ab00098fc061171914b38bcf01612442ce4777d52e74ba6a700896" exitCode=0
Oct 30 18:10:15.876893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:15.876458    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg" event={"ID":"f6a0aff1-9416-41ec-b272-d860b14371d2","Type":"ContainerDied","Data":"2806600eb9ab00098fc061171914b38bcf01612442ce4777d52e74ba6a700896"}
Oct 30 18:10:15.876893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:15.876518    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg" event={"ID":"f6a0aff1-9416-41ec-b272-d860b14371d2","Type":"ContainerStarted","Data":"fa8f840ea9f119b4392d584123c4f203cbe71f1af8cda4f18086b8fffb6572ca"}
Oct 30 18:10:17.026762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.026736    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:17.175272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.175183    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca\") pod \"f6a0aff1-9416-41ec-b272-d860b14371d2\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") "
Oct 30 18:10:17.175272 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.175244    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw\" (UniqueName: \"kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw\") pod \"f6a0aff1-9416-41ec-b272-d860b14371d2\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") "
Oct 30 18:10:17.175533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.175298    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca\") pod \"f6a0aff1-9416-41ec-b272-d860b14371d2\" (UID: \"f6a0aff1-9416-41ec-b272-d860b14371d2\") "
Oct 30 18:10:17.175609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.175572    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca" pod "f6a0aff1-9416-41ec-b272-d860b14371d2" (UID: "f6a0aff1-9416-41ec-b272-d860b14371d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:17.177539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.177513    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca" pod "f6a0aff1-9416-41ec-b272-d860b14371d2" (UID: "f6a0aff1-9416-41ec-b272-d860b14371d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:10:17.177632 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.177589    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw" pod "f6a0aff1-9416-41ec-b272-d860b14371d2" (UID: "f6a0aff1-9416-41ec-b272-d860b14371d2"). InnerVolumeSpecName "kube-api-access-mdvfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:17.276301 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.276261    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:17.276301 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.276296    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvfw\" (UniqueName: \"kubernetes.io/projected/f6a0aff1-9416-41ec-b272-d860b14371d2-kube-api-access-mdvfw\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:17.276301 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.276306    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a0aff1-9416-41ec-b272-d860b14371d2-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:17.304872 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.304834    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:17.890575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.890549    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"
Oct 30 18:10:17.890803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.890583    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg" event={"ID":"f6a0aff1-9416-41ec-b272-d860b14371d2","Type":"ContainerDied","Data":"fa8f840ea9f119b4392d584123c4f203cbe71f1af8cda4f18086b8fffb6572ca"}
Oct 30 18:10:17.890803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:17.890672    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa8f840ea9f119b4392d584123c4f203cbe71f1af8cda4f18086b8fffb6572ca"
Oct 30 18:10:18.848042 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:18.848009    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:19.028895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.028856    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"]
Oct 30 18:10:19.041589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.041562    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"]
Oct 30 18:10:19.044122 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.044096    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd69e8fe-9z6bb"]
Oct 30 18:10:19.045774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.045750    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20bcd8e4-9s9mw"]
Oct 30 18:10:19.085953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.085925    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:10:19.195707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.195614    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.195876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.195740    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.195876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.195787    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.195876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.195839    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.196056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.195899    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.196056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.195964    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.196056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196004    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.196056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196036    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.196288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196092    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token\") pod \"9355dd9f-4e3b-4584-a332-068331172693\" (UID: \"9355dd9f-4e3b-4584-a332-068331172693\") "
Oct 30 18:10:19.196288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196198    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:19.196288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:19.196465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196365    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:19.196594 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196484    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:19.196809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196617    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:19.196809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196624    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.196809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196676    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.196809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196692    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9355dd9f-4e3b-4584-a332-068331172693-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.196809 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.196703    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.198143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.198110    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:10:19.198292 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.198241    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:19.198515 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.198498    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:19.198515 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.198474    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6" pod "9355dd9f-4e3b-4584-a332-068331172693" (UID: "9355dd9f-4e3b-4584-a332-068331172693"). InnerVolumeSpecName "kube-api-access-nzqv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:19.298222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.298190    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqv6\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-kube-api-access-nzqv6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.298222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.298217    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.298222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.298228    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9355dd9f-4e3b-4584-a332-068331172693-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.298474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.298236    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9355dd9f-4e3b-4584-a332-068331172693-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.298474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.298245    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9355dd9f-4e3b-4584-a332-068331172693-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:19.907516 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.907469    5534 generic.go:356] "Generic (PLEG): container finished" podID="9355dd9f-4e3b-4584-a332-068331172693" containerID="ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a" exitCode=0
Oct 30 18:10:19.908014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.907537    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"
Oct 30 18:10:19.908014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.907549    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" event={"ID":"9355dd9f-4e3b-4584-a332-068331172693","Type":"ContainerDied","Data":"ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a"}
Oct 30 18:10:19.908014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.907617    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d" event={"ID":"9355dd9f-4e3b-4584-a332-068331172693","Type":"ContainerDied","Data":"98fabf6e8746d86449a6714924d8d67638d8d1adc3a599aaf4b170bca41617b9"}
Oct 30 18:10:19.908014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.907679    5534 scope.go:122] "RemoveContainer" containerID="ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a"
Oct 30 18:10:19.919590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.919569    5534 scope.go:122] "RemoveContainer" containerID="67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"
Oct 30 18:10:19.928882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.928862    5534 scope.go:122] "RemoveContainer" containerID="ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a"
Oct 30 18:10:19.929190 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:19.929170    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a\": container with ID starting with ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a not found: ID does not exist" containerID="ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a"
Oct 30 18:10:19.929240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.929199    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a"} err="failed to get container status \"ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a\": rpc error: code = NotFound desc = could not find container \"ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a\": container with ID starting with ae2aae9496d5d73b6cff648809b05468ffd6b08aa6723c0403f85ef79e27b06a not found: ID does not exist"
Oct 30 18:10:19.929240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.929217    5534 scope.go:122] "RemoveContainer" containerID="67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"
Oct 30 18:10:19.929477 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:19.929459    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7\": container with ID starting with 67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7 not found: ID does not exist" containerID="67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"
Oct 30 18:10:19.929543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.929483    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7"} err="failed to get container status \"67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7\": rpc error: code = NotFound desc = could not find container \"67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7\": container with ID starting with 67aa9367735ee4ddf44e4e867e73e97893c30a27379b03de1a68d86ada58aab7 not found: ID does not exist"
Oct 30 18:10:19.971180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.971143    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"]
Oct 30 18:10:19.976979 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:19.976948    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d323f5-openshift-default-d45c66f47-mct2d"]
Oct 30 18:10:20.371110 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.371078    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38600102-28ac-406e-a8e7-2d8c08533f55" path="/var/lib/kubelet/pods/38600102-28ac-406e-a8e7-2d8c08533f55/volumes"
Oct 30 18:10:20.371530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.371516    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9355dd9f-4e3b-4584-a332-068331172693" path="/var/lib/kubelet/pods/9355dd9f-4e3b-4584-a332-068331172693/volumes"
Oct 30 18:10:20.371944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.371929    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f27c89-9fdc-46cc-8d8e-ad9663892163" path="/var/lib/kubelet/pods/c0f27c89-9fdc-46cc-8d8e-ad9663892163/volumes"
Oct 30 18:10:20.576838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.576809    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:20.883818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.883772    5534 scope.go:122] "RemoveContainer" containerID="f1ab903ec789b27a03e7e416a010280993771ccba37053be26ca3fc711a5c793"
Oct 30 18:10:20.896715 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.896688    5534 scope.go:122] "RemoveContainer" containerID="74f086ed2866164ac6b842c7c90e939dd7960e90281fc2251c4e553c18c2167d"
Oct 30 18:10:20.907280 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.907255    5534 scope.go:122] "RemoveContainer" containerID="a4b8c9cbd9a203c07018a105e22812c601a76bac880308cf5456b7f33075f013"
Oct 30 18:10:20.918189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.918163    5534 scope.go:122] "RemoveContainer" containerID="6afa0145d56097871467cb799815a2c13e6f7821b64e9ebed8202d6ffb199555"
Oct 30 18:10:20.927322 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.927298    5534 scope.go:122] "RemoveContainer" containerID="0d9a60971c79154b746d173a649095eb70165eda26204175f1f6d68c4dd137b6"
Oct 30 18:10:20.936546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.936526    5534 scope.go:122] "RemoveContainer" containerID="6ee4aff4e0907d9bddb162238072ca56a91d7d98833d98189f5a2cfa309593a5"
Oct 30 18:10:20.945595 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:20.945576    5534 scope.go:122] "RemoveContainer" containerID="170b3e2855ee732e8ce29d41bd7e10434187ada24b428e91d2cfc76a9146848a"
Oct 30 18:10:22.071162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:22.071125    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:23.626816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:23.626784    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:24.796226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:24.796142    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"]
Oct 30 18:10:24.828511 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:24.828466    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"]
Oct 30 18:10:24.932822 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:24.932765    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:25.135807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:25.135773    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:25.247919 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:25.247870    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:26.776187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:26.776134    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:28.124330 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:28.124298    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:28.669997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:28.669939    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:28.784422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:28.784385    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:28.784668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:28.784622    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"]
Oct 30 18:10:28.796782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:28.796739    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a2a6cb-tx5zg"]
Oct 30 18:10:28.897123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:28.897076    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:29.326167 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:29.326116    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:29.796752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:29.796699    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" podUID="045cf88a-234e-43fa-936d-d139c5786225" containerName="payload-processing" containerID="cri-o://2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d" gracePeriod=25
Oct 30 18:10:29.829518 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:29.829456    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" podUID="d6c0b287-2026-4261-a085-073736c390c8" containerName="payload-pre-processing" containerID="cri-o://7ecc029fa2570a64f5ad64d2679d04fb3618dd6c91aabc0327a87ad661ce781d" gracePeriod=25
Oct 30 18:10:30.368457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:30.368416    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a0aff1-9416-41ec-b272-d860b14371d2" path="/var/lib/kubelet/pods/f6a0aff1-9416-41ec-b272-d860b14371d2/volumes"
Oct 30 18:10:30.502328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:30.502284    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:31.474813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:31.474778    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:32.093565 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:32.093528    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:32.355727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:32.355626    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:33.773620 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:33.773590    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:35.304958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:35.304917    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:35.419607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:35.419577    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:35.653106 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:35.653071    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"]
Oct 30 18:10:35.653362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:35.653333    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy" containerID="cri-o://a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1" gracePeriod=30
Oct 30 18:10:35.694961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:35.694914    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:37.047454 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:37.047417    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:37.580898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:37.580846    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" podUID="d6c0b287-2026-4261-a085-073736c390c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:10:37.580898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:37.580883    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" podUID="045cf88a-234e-43fa-936d-d139c5786225" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:10:38.664114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:38.664073    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:40.133052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:40.133019    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:40.926066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:40.926004    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 18:10:41.010727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010699    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.010937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010749    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.010937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010807    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.010937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010834    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.010937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010863    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.010937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010898    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.011219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.010945    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.011219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011012    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.011219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011007    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:41.011219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011041    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks\") pod \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\" (UID: \"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9\") "
Oct 30 18:10:41.011428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011413    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.011428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011043    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:41.011532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011282    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:41.011679 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.011634    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:41.012398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.012372    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:41.013411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.013390    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:10:41.013735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.013708    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "kube-api-access-xvlks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:41.013820 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.013768    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:41.014716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.014692    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy" pod "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" (UID: "cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:10:41.041138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.041102    5534 generic.go:356] "Generic (PLEG): container finished" podID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerID="a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1" exitCode=0
Oct 30 18:10:41.041333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.041148    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"
Oct 30 18:10:41.041333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.041183    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" event={"ID":"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9","Type":"ContainerDied","Data":"a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1"}
Oct 30 18:10:41.041333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.041243    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r" event={"ID":"cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9","Type":"ContainerDied","Data":"b26625f6d95c3dbe8b849daede36ec1da295f3c2fc835523ca240de62c60067b"}
Oct 30 18:10:41.041333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.041266    5534 scope.go:122] "RemoveContainer" containerID="a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1"
Oct 30 18:10:41.052524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.052507    5534 scope.go:122] "RemoveContainer" containerID="ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"
Oct 30 18:10:41.062068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.062044    5534 scope.go:122] "RemoveContainer" containerID="a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1"
Oct 30 18:10:41.062338 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:41.062306    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1\": container with ID starting with a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1 not found: ID does not exist" containerID="a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1"
Oct 30 18:10:41.062420 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.062351    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1"} err="failed to get container status \"a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1\": rpc error: code = NotFound desc = could not find container \"a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1\": container with ID starting with a67ce84228ef6d9ea264094f32f005127ee7fb4fcf5a86c962d078bb02e3e7f1 not found: ID does not exist"
Oct 30 18:10:41.062420 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.062369    5534 scope.go:122] "RemoveContainer" containerID="ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"
Oct 30 18:10:41.062692 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:41.062666    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5\": container with ID starting with ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5 not found: ID does not exist" containerID="ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"
Oct 30 18:10:41.062783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.062702    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5"} err="failed to get container status \"ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5\": rpc error: code = NotFound desc = could not find container \"ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5\": container with ID starting with ab31f078eabe6eef1195200dbc38f81e36d71201eadb89cbf9437864c87333c5 not found: ID does not exist"
Oct 30 18:10:41.096720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.096683    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"]
Oct 30 18:10:41.100325 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.100299    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-xfr8r"]
Oct 30 18:10:41.112157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112132    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112159    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlks\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-kube-api-access-xvlks\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112171    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112180    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112188    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112197    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112204    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.112245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.112212    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:41.701003 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:41.700969    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:42.368869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:42.368831    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" path="/var/lib/kubelet/pods/cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9/volumes"
Oct 30 18:10:43.388692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:43.388658    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:44.852811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:44.852773    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:45.243596 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:45.243502    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:10:46.324874 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:46.324844    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:46.750570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:46.750537    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:46.865409 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:46.865376    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:47.581068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.581025    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" podUID="045cf88a-234e-43fa-936d-d139c5786225" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:10:47.581534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.581025    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" podUID="d6c0b287-2026-4261-a085-073736c390c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:10:47.624947 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.624913    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"]
Oct 30 18:10:47.625451 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.625411    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7" podUID="6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" containerName="maas-api" containerID="cri-o://44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0" gracePeriod=30
Oct 30 18:10:47.696937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.696904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:47.906958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.906933    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 18:10:47.979190 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.979140    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n\" (UniqueName: \"kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n\") pod \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") "
Oct 30 18:10:47.979391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.979208    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs\") pod \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") "
Oct 30 18:10:47.979391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.979275    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls\") pod \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") "
Oct 30 18:10:47.979541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.979418    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca\") pod \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\" (UID: \"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd\") "
Oct 30 18:10:47.980098 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.980071    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca" pod "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" (UID: "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:47.981752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.981712    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n" pod "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" (UID: "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd"). InnerVolumeSpecName "kube-api-access-rz87n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:47.982218 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.982197    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs" pod "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" (UID: "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:10:47.982298 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:47.982254    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls" pod "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" (UID: "6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:10:48.081007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.080972    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:48.081007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.081009    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz87n\" (UniqueName: \"kubernetes.io/projected/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-kube-api-access-rz87n\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:48.081251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.081026    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:48.081251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.081041    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:48.086511 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.086479    5534 generic.go:356] "Generic (PLEG): container finished" podID="6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" containerID="44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0" exitCode=0
Oct 30 18:10:48.086705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.086539    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"
Oct 30 18:10:48.086705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.086554    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7" event={"ID":"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd","Type":"ContainerDied","Data":"44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0"}
Oct 30 18:10:48.086705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.086632    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7" event={"ID":"6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd","Type":"ContainerDied","Data":"9a9a14595f91d051c3bf72ec391fd9ddd5d2ebf242808124ba55fc25665fe6da"}
Oct 30 18:10:48.086705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.086703    5534 scope.go:122] "RemoveContainer" containerID="44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0"
Oct 30 18:10:48.097564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.097542    5534 scope.go:122] "RemoveContainer" containerID="44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0"
Oct 30 18:10:48.097989 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:48.097969    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0\": container with ID starting with 44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0 not found: ID does not exist" containerID="44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0"
Oct 30 18:10:48.098061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.097997    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0"} err="failed to get container status \"44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0\": rpc error: code = NotFound desc = could not find container \"44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0\": container with ID starting with 44dfd6971c0b58d2abc82007752e43538982a44a84d6c8e535098ce035467ba0 not found: ID does not exist"
Oct 30 18:10:48.138800 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.138715    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"]
Oct 30 18:10:48.142991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.142966    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a23c98-5f4c867476-cp6m7"]
Oct 30 18:10:48.278344 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.278314    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:48.368481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.368445    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" path="/var/lib/kubelet/pods/6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd/volumes"
Oct 30 18:10:48.412527 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.412446    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:48.884536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:48.884478    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:49.915540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:49.915481    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:50.030549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.030513    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"]
Oct 30 18:10:50.042194 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.042164    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"]
Oct 30 18:10:50.043357 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.043340    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c35a00-g2kc8"]
Oct 30 18:10:50.044493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.044469    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-018aff3d-gzdtd"]
Oct 30 18:10:50.084135 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.084107    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 18:10:50.102339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.102311    5534 generic.go:356] "Generic (PLEG): container finished" podID="045cf88a-234e-43fa-936d-d139c5786225" containerID="2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d" exitCode=0
Oct 30 18:10:50.102482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.102359    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"
Oct 30 18:10:50.102482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.102413    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" event={"ID":"045cf88a-234e-43fa-936d-d139c5786225","Type":"ContainerDied","Data":"2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d"}
Oct 30 18:10:50.102597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.102497    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc" event={"ID":"045cf88a-234e-43fa-936d-d139c5786225","Type":"ContainerDied","Data":"d98e5a2ecdf3cf1410ddc74482f6ffea9b827131175ff8f604e6b723cb7f0c1c"}
Oct 30 18:10:50.102597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.102529    5534 scope.go:122] "RemoveContainer" containerID="2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d"
Oct 30 18:10:50.104314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.104290    5534 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b287-2026-4261-a085-073736c390c8" containerID="7ecc029fa2570a64f5ad64d2679d04fb3618dd6c91aabc0327a87ad661ce781d" exitCode=0
Oct 30 18:10:50.104478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.104363    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" event={"ID":"d6c0b287-2026-4261-a085-073736c390c8","Type":"ContainerDied","Data":"7ecc029fa2570a64f5ad64d2679d04fb3618dd6c91aabc0327a87ad661ce781d"}
Oct 30 18:10:50.119717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.119703    5534 scope.go:122] "RemoveContainer" containerID="2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d"
Oct 30 18:10:50.119980 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:10:50.119962    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d\": container with ID starting with 2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d not found: ID does not exist" containerID="2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d"
Oct 30 18:10:50.120032 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.119991    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d"} err="failed to get container status \"2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d\": rpc error: code = NotFound desc = could not find container \"2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d\": container with ID starting with 2d563e48d0e668190f3cbd4106e8b78c616cd616ec3d7628b78ac1638bb9f76d not found: ID does not exist"
Oct 30 18:10:50.122581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.122563    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 18:10:50.203993 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.203955    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb\" (UniqueName: \"kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb\") pod \"d6c0b287-2026-4261-a085-073736c390c8\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") "
Oct 30 18:10:50.204194 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204021    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume\") pod \"045cf88a-234e-43fa-936d-d139c5786225\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") "
Oct 30 18:10:50.204194 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204117    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b\" (UniqueName: \"kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b\") pod \"045cf88a-234e-43fa-936d-d139c5786225\" (UID: \"045cf88a-234e-43fa-936d-d139c5786225\") "
Oct 30 18:10:50.204342 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204242    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume\") pod \"d6c0b287-2026-4261-a085-073736c390c8\" (UID: \"d6c0b287-2026-4261-a085-073736c390c8\") "
Oct 30 18:10:50.204406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204389    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume" pod "045cf88a-234e-43fa-936d-d139c5786225" (UID: "045cf88a-234e-43fa-936d-d139c5786225"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:50.204618 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204597    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume" pod "d6c0b287-2026-4261-a085-073736c390c8" (UID: "d6c0b287-2026-4261-a085-073736c390c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:50.204762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204745    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0b287-2026-4261-a085-073736c390c8-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:50.204837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.204764    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045cf88a-234e-43fa-936d-d139c5786225-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:50.206886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.206858    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb" pod "d6c0b287-2026-4261-a085-073736c390c8" (UID: "d6c0b287-2026-4261-a085-073736c390c8"). InnerVolumeSpecName "kube-api-access-jgsdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:50.206886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.206875    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b" pod "045cf88a-234e-43fa-936d-d139c5786225" (UID: "045cf88a-234e-43fa-936d-d139c5786225"). InnerVolumeSpecName "kube-api-access-7gr2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:50.305509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.305467    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gr2b\" (UniqueName: \"kubernetes.io/projected/045cf88a-234e-43fa-936d-d139c5786225-kube-api-access-7gr2b\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:50.305509 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.305508    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsdb\" (UniqueName: \"kubernetes.io/projected/d6c0b287-2026-4261-a085-073736c390c8-kube-api-access-jgsdb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:50.369527 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.369482    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63b2055-6998-4ada-8b98-6986ef43ecc0" path="/var/lib/kubelet/pods/f63b2055-6998-4ada-8b98-6986ef43ecc0/volumes"
Oct 30 18:10:50.370025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.370003    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff702ec3-0e0d-4cbc-b57e-e5f10437634e" path="/var/lib/kubelet/pods/ff702ec3-0e0d-4cbc-b57e-e5f10437634e/volumes"
Oct 30 18:10:50.440117 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.440086    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"]
Oct 30 18:10:50.451944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:50.451911    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a2a6cb-6d7b7968f8-z55dc"]
Oct 30 18:10:51.111718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:51.111688    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"
Oct 30 18:10:51.111718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:51.111708    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7" event={"ID":"d6c0b287-2026-4261-a085-073736c390c8","Type":"ContainerDied","Data":"72870bdf3c5a8871307f7e47c5760bad7d33ea905a90838327751ff111b6ef62"}
Oct 30 18:10:51.112347 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:51.111771    5534 scope.go:122] "RemoveContainer" containerID="7ecc029fa2570a64f5ad64d2679d04fb3618dd6c91aabc0327a87ad661ce781d"
Oct 30 18:10:51.160968 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:51.160929    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"]
Oct 30 18:10:51.172941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:51.172902    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a2a6cb-7985494dcf-mb7g7"]
Oct 30 18:10:51.407798 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:51.407705    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:52.368323 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:52.368281    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045cf88a-234e-43fa-936d-d139c5786225" path="/var/lib/kubelet/pods/045cf88a-234e-43fa-936d-d139c5786225/volumes"
Oct 30 18:10:52.368916 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:52.368895    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c0b287-2026-4261-a085-073736c390c8" path="/var/lib/kubelet/pods/d6c0b287-2026-4261-a085-073736c390c8/volumes"
Oct 30 18:10:52.937622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:52.937594    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:54.413427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:54.413395    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:55.317982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:55.317949    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:55.429116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:55.429079    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:55.658675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:55.658606    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:56.050858 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.050768    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:56.170912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.170871    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:56.272338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272302    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"]
Oct 30 18:10:56.272728 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272711    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272729    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272747    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d6c0b287-2026-4261-a085-073736c390c8" containerName="payload-pre-processing"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272753    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c0b287-2026-4261-a085-073736c390c8" containerName="payload-pre-processing"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272762    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272767    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272775    5534 cpu_manager.go:414] "containerMap: removing container" podUID="045cf88a-234e-43fa-936d-d139c5786225" containerName="payload-processing"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272781    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045cf88a-234e-43fa-936d-d139c5786225" containerName="payload-processing"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272793    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272797    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272812    5534 cpu_manager.go:414] "containerMap: removing container" podUID="6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" containerName="maas-api"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272820    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" containerName="maas-api"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272829    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272834    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272847    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f6a0aff1-9416-41ec-b272-d860b14371d2" containerName="revoke-keys"
Oct 30 18:10:56.272828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272852    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a0aff1-9416-41ec-b272-d860b14371d2" containerName="revoke-keys"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272915    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="6462b7fa-fcbd-4c93-a2b7-45fc9b722fdd" containerName="maas-api"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272925    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272933    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a0aff1-9416-41ec-b272-d860b14371d2" containerName="revoke-keys"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272940    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c688455a-36a2-4f26-9ccd-20be9f3af129" containerName="istio-proxy"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272946    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272951    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c0b287-2026-4261-a085-073736c390c8" containerName="payload-pre-processing"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272957    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="045cf88a-234e-43fa-936d-d139c5786225" containerName="payload-processing"
Oct 30 18:10:56.273818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.272968    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9355dd9f-4e3b-4584-a332-068331172693" containerName="istio-proxy"
Oct 30 18:10:56.277469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.277446    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.294219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.294190    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:10:56.296367 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.296330    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"]
Oct 30 18:10:56.306131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.306059    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:10:56.306312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.306105    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:10:56.306425 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.306227    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:10:56.366773 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.366745    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.366960 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.366782    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.366960 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.366886    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrt9\" (UniqueName: \"kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.467690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.467569    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.468434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.468402    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.468617 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.468538    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrt9\" (UniqueName: \"kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.468617 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.468546    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.471092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.471061    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.492270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.492233    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrt9\" (UniqueName: \"kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9\") pod \"maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.591291 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.591257    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:56.894679 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:56.894629    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"]
Oct 30 18:10:56.897350 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:10:56.897322    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef04a8c_7415_44d4_b338_31da778eb806.slice/crio-5096e1b32ada410c1e15b2d9c509aed09c232b10dae812e800cfc87320f0edcd WatchSource:0}: Error finding container 5096e1b32ada410c1e15b2d9c509aed09c232b10dae812e800cfc87320f0edcd: Status 404 returned error can't find the container with id 5096e1b32ada410c1e15b2d9c509aed09c232b10dae812e800cfc87320f0edcd
Oct 30 18:10:57.150156 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:57.150073    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l" event={"ID":"bef04a8c-7415-44d4-b338-31da778eb806","Type":"ContainerStarted","Data":"5096e1b32ada410c1e15b2d9c509aed09c232b10dae812e800cfc87320f0edcd"}
Oct 30 18:10:57.760287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:57.760253    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:58.157607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:58.157569    5534 generic.go:356] "Generic (PLEG): container finished" podID="bef04a8c-7415-44d4-b338-31da778eb806" containerID="0910890d5f755f08e5439dfe26e1e3de2a68f941eaf6a12d721854ac6684fc19" exitCode=0
Oct 30 18:10:58.157803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:58.157607    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l" event={"ID":"bef04a8c-7415-44d4-b338-31da778eb806","Type":"ContainerDied","Data":"0910890d5f755f08e5439dfe26e1e3de2a68f941eaf6a12d721854ac6684fc19"}
Oct 30 18:10:59.297614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.297582    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:10:59.326578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.326547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:10:59.395166 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.395135    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca\") pod \"bef04a8c-7415-44d4-b338-31da778eb806\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") "
Oct 30 18:10:59.395166 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.395170    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca\") pod \"bef04a8c-7415-44d4-b338-31da778eb806\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") "
Oct 30 18:10:59.395442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.395199    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9\" (UniqueName: \"kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9\") pod \"bef04a8c-7415-44d4-b338-31da778eb806\" (UID: \"bef04a8c-7415-44d4-b338-31da778eb806\") "
Oct 30 18:10:59.395517 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.395455    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca" pod "bef04a8c-7415-44d4-b338-31da778eb806" (UID: "bef04a8c-7415-44d4-b338-31da778eb806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:10:59.397406 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.397380    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca" pod "bef04a8c-7415-44d4-b338-31da778eb806" (UID: "bef04a8c-7415-44d4-b338-31da778eb806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:10:59.397527 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.397439    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9" pod "bef04a8c-7415-44d4-b338-31da778eb806" (UID: "bef04a8c-7415-44d4-b338-31da778eb806"). InnerVolumeSpecName "kube-api-access-gqrt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:10:59.496708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.496592    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:59.496708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.496621    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef04a8c-7415-44d4-b338-31da778eb806-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:10:59.496708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:10:59.496630    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrt9\" (UniqueName: \"kubernetes.io/projected/bef04a8c-7415-44d4-b338-31da778eb806-kube-api-access-gqrt9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:00.169686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:00.169636    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l" event={"ID":"bef04a8c-7415-44d4-b338-31da778eb806","Type":"ContainerDied","Data":"5096e1b32ada410c1e15b2d9c509aed09c232b10dae812e800cfc87320f0edcd"}
Oct 30 18:11:00.169686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:00.169690    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"
Oct 30 18:11:00.169910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:00.169699    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5096e1b32ada410c1e15b2d9c509aed09c232b10dae812e800cfc87320f0edcd"
Oct 30 18:11:00.791150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:00.791122    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:01.076495 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:01.076414    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"]
Oct 30 18:11:01.184925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:01.184896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:02.492288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:02.492238    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:03.954954 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:03.954905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:04.611882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:04.611805    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:05.513833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:05.513777    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:06.869028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:06.868991    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:06.980616 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:06.980584    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:07.102063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:07.102033    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:08.434256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.434219    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"]
Oct 30 18:11:08.434762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.434677    5534 cpu_manager.go:414] "containerMap: removing container" podUID="bef04a8c-7415-44d4-b338-31da778eb806" containerName="revoke-keys"
Oct 30 18:11:08.434762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.434693    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef04a8c-7415-44d4-b338-31da778eb806" containerName="revoke-keys"
Oct 30 18:11:08.434762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.434763    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef04a8c-7415-44d4-b338-31da778eb806" containerName="revoke-keys"
Oct 30 18:11:08.434896 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.434772    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd9f802-7d3b-45e6-9e0f-447ae8a11fa9" containerName="istio-proxy"
Oct 30 18:11:08.438195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.438175    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.449854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.449834    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"]
Oct 30 18:11:08.453283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.453266    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.459808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.459617    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:11:08.459808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.459782    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:11:08.468427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.468407    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:11:08.468536 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.468442    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:11:08.470034 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.470016    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"]
Oct 30 18:11:08.486624 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.486578    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"]
Oct 30 18:11:08.583055 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.583022    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4b7\" (UniqueName: \"kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.583246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.583076    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.583246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.583162    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.583246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.583199    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.583246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.583231    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqbv\" (UniqueName: \"kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.583401 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.583330    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.684123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684086    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.684336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684174    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4b7\" (UniqueName: \"kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.684336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684231    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.684336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684280    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.684336 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684305    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.684577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684347    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqbv\" (UniqueName: \"kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.684961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.684934    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.685149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.685126    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.686906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.686879    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.687025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.687007    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.707160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.707113    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4b7\" (UniqueName: \"kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7\") pod \"maas-api-revoke-sub-c0c7158d-8wdjn\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.707320 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.707125    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqbv\" (UniqueName: \"kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv\") pod \"maas-api-revoke-sub-b646be53-b9rp5\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:08.752192 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.752155    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:08.769631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:08.769597    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:09.056124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:09.056100    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"]
Oct 30 18:11:09.058547 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:11:09.058518    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01deb567_0ca4_47f9_8118_b26b478a4226.slice/crio-61eacdcf82b5917641cd375389d3fa5f9db0ed3ad652b85d14db524e19ece8e5 WatchSource:0}: Error finding container 61eacdcf82b5917641cd375389d3fa5f9db0ed3ad652b85d14db524e19ece8e5: Status 404 returned error can't find the container with id 61eacdcf82b5917641cd375389d3fa5f9db0ed3ad652b85d14db524e19ece8e5
Oct 30 18:11:09.080840 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:09.080818    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"]
Oct 30 18:11:09.091912 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:11:09.091883    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb76301d_6e71_4ae3_9dbd_dc58de3d81e6.slice/crio-9d68871307a2462b3181e0cbb33a1a44e295cd449d97d6e33efc1a1c3a907c32 WatchSource:0}: Error finding container 9d68871307a2462b3181e0cbb33a1a44e295cd449d97d6e33efc1a1c3a907c32: Status 404 returned error can't find the container with id 9d68871307a2462b3181e0cbb33a1a44e295cd449d97d6e33efc1a1c3a907c32
Oct 30 18:11:09.229175 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:09.229136    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5" event={"ID":"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6","Type":"ContainerStarted","Data":"9d68871307a2462b3181e0cbb33a1a44e295cd449d97d6e33efc1a1c3a907c32"}
Oct 30 18:11:09.230474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:09.230443    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn" event={"ID":"01deb567-0ca4-47f9-8118-b26b478a4226","Type":"ContainerStarted","Data":"61eacdcf82b5917641cd375389d3fa5f9db0ed3ad652b85d14db524e19ece8e5"}
Oct 30 18:11:10.237539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:10.237455    5534 generic.go:356] "Generic (PLEG): container finished" podID="01deb567-0ca4-47f9-8118-b26b478a4226" containerID="8a9ebd76515303d2d75aa1009a4b12e5d5a1c09872cedda4ccfa777d32bcc3e7" exitCode=0
Oct 30 18:11:10.238000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:10.237536    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn" event={"ID":"01deb567-0ca4-47f9-8118-b26b478a4226","Type":"ContainerDied","Data":"8a9ebd76515303d2d75aa1009a4b12e5d5a1c09872cedda4ccfa777d32bcc3e7"}
Oct 30 18:11:10.239249 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:10.239222    5534 generic.go:356] "Generic (PLEG): container finished" podID="fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" containerID="1898fe33ef7626587c327e5eeb49a5a337651eb888c3a92196bb697508e4ed63" exitCode=0
Oct 30 18:11:10.239381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:10.239307    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5" event={"ID":"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6","Type":"ContainerDied","Data":"1898fe33ef7626587c327e5eeb49a5a337651eb888c3a92196bb697508e4ed63"}
Oct 30 18:11:11.395465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.395438    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:11.398458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.398440    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:11.514395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514313    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca\") pod \"01deb567-0ca4-47f9-8118-b26b478a4226\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") "
Oct 30 18:11:11.514395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514356    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca\") pod \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") "
Oct 30 18:11:11.514626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514449    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv\" (UniqueName: \"kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv\") pod \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") "
Oct 30 18:11:11.514626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514478    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7\" (UniqueName: \"kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7\") pod \"01deb567-0ca4-47f9-8118-b26b478a4226\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") "
Oct 30 18:11:11.514626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514515    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca\") pod \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\" (UID: \"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6\") "
Oct 30 18:11:11.514626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514561    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca\") pod \"01deb567-0ca4-47f9-8118-b26b478a4226\" (UID: \"01deb567-0ca4-47f9-8118-b26b478a4226\") "
Oct 30 18:11:11.514899 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514782    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca" pod "fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" (UID: "fb76301d-6e71-4ae3-9dbd-dc58de3d81e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:11:11.515007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.514985    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:11.515287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.515224    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca" pod "01deb567-0ca4-47f9-8118-b26b478a4226" (UID: "01deb567-0ca4-47f9-8118-b26b478a4226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:11:11.516977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.516939    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv" pod "fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" (UID: "fb76301d-6e71-4ae3-9dbd-dc58de3d81e6"). InnerVolumeSpecName "kube-api-access-rwqbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:11:11.517085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.516986    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca" pod "01deb567-0ca4-47f9-8118-b26b478a4226" (UID: "01deb567-0ca4-47f9-8118-b26b478a4226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:11:11.517085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.517003    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7" pod "01deb567-0ca4-47f9-8118-b26b478a4226" (UID: "01deb567-0ca4-47f9-8118-b26b478a4226"). InnerVolumeSpecName "kube-api-access-5w4b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:11:11.517219 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.517122    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca" pod "fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" (UID: "fb76301d-6e71-4ae3-9dbd-dc58de3d81e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:11:11.616505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.616476    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:11.616505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.616502    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqbv\" (UniqueName: \"kubernetes.io/projected/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-kube-api-access-rwqbv\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:11.616700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.616511    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4b7\" (UniqueName: \"kubernetes.io/projected/01deb567-0ca4-47f9-8118-b26b478a4226-kube-api-access-5w4b7\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:11.616700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.616521    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:11.616700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:11.616530    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01deb567-0ca4-47f9-8118-b26b478a4226-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:12.253247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:12.253220    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"
Oct 30 18:11:12.253442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:12.253247    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5" event={"ID":"fb76301d-6e71-4ae3-9dbd-dc58de3d81e6","Type":"ContainerDied","Data":"9d68871307a2462b3181e0cbb33a1a44e295cd449d97d6e33efc1a1c3a907c32"}
Oct 30 18:11:12.253442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:12.253302    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d68871307a2462b3181e0cbb33a1a44e295cd449d97d6e33efc1a1c3a907c32"
Oct 30 18:11:12.254771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:12.254747    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"
Oct 30 18:11:12.254771 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:12.254755    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn" event={"ID":"01deb567-0ca4-47f9-8118-b26b478a4226","Type":"ContainerDied","Data":"61eacdcf82b5917641cd375389d3fa5f9db0ed3ad652b85d14db524e19ece8e5"}
Oct 30 18:11:12.255009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:12.254814    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61eacdcf82b5917641cd375389d3fa5f9db0ed3ad652b85d14db524e19ece8e5"
Oct 30 18:11:15.243925 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:11:15.243878    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:11:16.097006 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.096968    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main" containerID="cri-o://0a9047705dedf7f24614e7335288740731f8a31a7929e551a26c8acea79e1db9" gracePeriod=45
Oct 30 18:11:16.279748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.279696    5534 generic.go:356] "Generic (PLEG): container finished" podID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerID="0a9047705dedf7f24614e7335288740731f8a31a7929e551a26c8acea79e1db9" exitCode=0
Oct 30 18:11:16.280114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.279767    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" event={"ID":"4c768139-2889-4936-b9bd-34fd5d0718fd","Type":"ContainerDied","Data":"0a9047705dedf7f24614e7335288740731f8a31a7929e551a26c8acea79e1db9"}
Oct 30 18:11:16.349002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.348938    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 18:11:16.359060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359036    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq\" (UniqueName: \"kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq\") pod \"4c768139-2889-4936-b9bd-34fd5d0718fd\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") "
Oct 30 18:11:16.359186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359069    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir\") pod \"4c768139-2889-4936-b9bd-34fd5d0718fd\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") "
Oct 30 18:11:16.359186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359138    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm\") pod \"4c768139-2889-4936-b9bd-34fd5d0718fd\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") "
Oct 30 18:11:16.359186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359168    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache\") pod \"4c768139-2889-4936-b9bd-34fd5d0718fd\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") "
Oct 30 18:11:16.359372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359201    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home\") pod \"4c768139-2889-4936-b9bd-34fd5d0718fd\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") "
Oct 30 18:11:16.359372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359224    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs\") pod \"4c768139-2889-4936-b9bd-34fd5d0718fd\" (UID: \"4c768139-2889-4936-b9bd-34fd5d0718fd\") "
Oct 30 18:11:16.359484 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359381    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir" pod "4c768139-2889-4936-b9bd-34fd5d0718fd" (UID: "4c768139-2889-4936-b9bd-34fd5d0718fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:16.359484 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359463    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home" pod "4c768139-2889-4936-b9bd-34fd5d0718fd" (UID: "4c768139-2889-4936-b9bd-34fd5d0718fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:16.359569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359476    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache" pod "4c768139-2889-4936-b9bd-34fd5d0718fd" (UID: "4c768139-2889-4936-b9bd-34fd5d0718fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:16.359569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.359528    5534 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-tmp-dir\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:16.361422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.361397    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs" pod "4c768139-2889-4936-b9bd-34fd5d0718fd" (UID: "4c768139-2889-4936-b9bd-34fd5d0718fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:11:16.361535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.361445    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm" pod "4c768139-2889-4936-b9bd-34fd5d0718fd" (UID: "4c768139-2889-4936-b9bd-34fd5d0718fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:16.361535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.361449    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq" pod "4c768139-2889-4936-b9bd-34fd5d0718fd" (UID: "4c768139-2889-4936-b9bd-34fd5d0718fd"). InnerVolumeSpecName "kube-api-access-k69hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:11:16.460806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.460774    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69hq\" (UniqueName: \"kubernetes.io/projected/4c768139-2889-4936-b9bd-34fd5d0718fd-kube-api-access-k69hq\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:16.460806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.460804    5534 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-dshm\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:16.461032 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.460822    5534 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-model-cache\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:16.461032 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.460835    5534 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c768139-2889-4936-b9bd-34fd5d0718fd-home\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:16.461032 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:16.460848    5534 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c768139-2889-4936-b9bd-34fd5d0718fd-tls-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:17.286589 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:17.286560    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"
Oct 30 18:11:17.287094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:17.286606    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp" event={"ID":"4c768139-2889-4936-b9bd-34fd5d0718fd","Type":"ContainerDied","Data":"82d74f314dd0f7df352beab1b9081df6cbf1f13820ab0011ac70f5df015779da"}
Oct 30 18:11:17.287094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:17.286677    5534 scope.go:122] "RemoveContainer" containerID="0a9047705dedf7f24614e7335288740731f8a31a7929e551a26c8acea79e1db9"
Oct 30 18:11:17.337141 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:17.337117    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"]
Oct 30 18:11:17.348755 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:17.348731    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-w4-0e7403-kserve-588c78b8d8-kjzlp"]
Oct 30 18:11:18.367963 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:18.367922    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" path="/var/lib/kubelet/pods/4c768139-2889-4936-b9bd-34fd5d0718fd/volumes"
Oct 30 18:11:21.010296 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.010263    5534 scope.go:122] "RemoveContainer" containerID="8fb7c048e14f211e38fea92328f36a50025a8d3d7c4a71a255b3b6e0b0dc1edb"
Oct 30 18:11:21.022009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.021981    5534 scope.go:122] "RemoveContainer" containerID="6514ed7161b8a46f3cbcccd39311190172cec4260731554a1cb01753b5e85e76"
Oct 30 18:11:21.031022 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.031000    5534 scope.go:122] "RemoveContainer" containerID="075a2f510cd27d7195d2bf88683b493731ba25c241a17b27aacdb76a45aa0a9b"
Oct 30 18:11:21.042467 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.042438    5534 scope.go:122] "RemoveContainer" containerID="b7b6f3e1cfdae40d9082cbc3c5a6e6167d68c747e4415ecda53ca164aa46f09d"
Oct 30 18:11:21.051470 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.051445    5534 scope.go:122] "RemoveContainer" containerID="9530580b8cac0e8d49726d8c0edf715a3b1ec73a332450efbd09e531828f584d"
Oct 30 18:11:21.061205 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.061184    5534 scope.go:122] "RemoveContainer" containerID="e3243778c9eb4796b376e0dc1eb71907c61518474e00211aa6cb97e844869d95"
Oct 30 18:11:21.071829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.071800    5534 scope.go:122] "RemoveContainer" containerID="d20c6ce7beeb471a3440bc04f7047174ec4f084a8c601bb0eb8c5a295a8f5946"
Oct 30 18:11:21.081866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:21.081841    5534 scope.go:122] "RemoveContainer" containerID="d351b1d826dc3e733d165fff36938b6dda2e18e20f5eb278edcccd62642696fd"
Oct 30 18:11:23.613184 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:23.613154    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:24.075698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:24.075586    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:24.145527 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:24.145492    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"]
Oct 30 18:11:24.157010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:24.156983    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0e7403-7sm6l"]
Oct 30 18:11:24.211451 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:24.211424    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:24.382801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:24.379931    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef04a8c-7415-44d4-b338-31da778eb806" path="/var/lib/kubelet/pods/bef04a8c-7415-44d4-b338-31da778eb806/volumes"
Oct 30 18:11:24.664162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:24.664033    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:30.773250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:30.773220    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:30.890305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:30.890272    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:31.072760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:31.072668    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"]
Oct 30 18:11:31.072939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:31.072919    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" containerID="cri-o://331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766" gracePeriod=30
Oct 30 18:11:31.141834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:31.141807    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:31.540533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:31.540504    5534 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:11:31.540715 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:31.540554    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:11:33.362982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:33.362948    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:36.090743 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.090706    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bb675f547-6kmcm"]
Oct 30 18:11:36.091290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.090981    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5bb675f547-6kmcm" podUID="9f9faf67-3b4f-454a-a7cc-6d95fea1997b" containerName="manager" containerID="cri-o://4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848" gracePeriod=30
Oct 30 18:11:36.365713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.365677    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 18:11:36.397596 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.397518    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 18:11:36.398818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.398684    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" event={"ID":"cab480c1-9f5e-4933-968f-7b3e9feb0b75","Type":"ContainerDied","Data":"331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766"}
Oct 30 18:11:36.398818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.398703    5534 generic.go:356] "Generic (PLEG): container finished" podID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerID="331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766" exitCode=0
Oct 30 18:11:36.398818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.398749    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p" event={"ID":"cab480c1-9f5e-4933-968f-7b3e9feb0b75","Type":"ContainerDied","Data":"7b4db08c5011be9574e9cfb79dd672fa1c4b32aed031f54fb1b71045dc3952e6"}
Oct 30 18:11:36.398818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.398694    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"
Oct 30 18:11:36.398818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.398779    5534 scope.go:122] "RemoveContainer" containerID="331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766"
Oct 30 18:11:36.400189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.400166    5534 generic.go:356] "Generic (PLEG): container finished" podID="9f9faf67-3b4f-454a-a7cc-6d95fea1997b" containerID="4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848" exitCode=0
Oct 30 18:11:36.400313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.400208    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-6kmcm"
Oct 30 18:11:36.400313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.400293    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-6kmcm" event={"ID":"9f9faf67-3b4f-454a-a7cc-6d95fea1997b","Type":"ContainerDied","Data":"4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848"}
Oct 30 18:11:36.400419 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.400333    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-6kmcm" event={"ID":"9f9faf67-3b4f-454a-a7cc-6d95fea1997b","Type":"ContainerDied","Data":"61532881003f7784edb8bab4c492c5c28e98f551e839705188997813060db4c2"}
Oct 30 18:11:36.408884 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.408867    5534 scope.go:122] "RemoveContainer" containerID="331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766"
Oct 30 18:11:36.409195 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:11:36.409165    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766\": container with ID starting with 331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766 not found: ID does not exist" containerID="331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766"
Oct 30 18:11:36.409275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.409205    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766"} err="failed to get container status \"331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766\": rpc error: code = NotFound desc = could not find container \"331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766\": container with ID starting with 331001c1ebfd238c39f1587de29f7ff6fcc94ba6c2ede8761aea9815b6c95766 not found: ID does not exist"
Oct 30 18:11:36.409275 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.409227    5534 scope.go:122] "RemoveContainer" containerID="4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848"
Oct 30 18:11:36.428060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.428033    5534 scope.go:122] "RemoveContainer" containerID="4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848"
Oct 30 18:11:36.428410 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:11:36.428384    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848\": container with ID starting with 4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848 not found: ID does not exist" containerID="4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848"
Oct 30 18:11:36.428471 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.428425    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848"} err="failed to get container status \"4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848\": rpc error: code = NotFound desc = could not find container \"4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848\": container with ID starting with 4cc50290b33cb31c01e69b9b0adeb6b2f88e39e363820c445f93463db8d70848 not found: ID does not exist"
Oct 30 18:11:36.453259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453230    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453272    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453298    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx\" (UniqueName: \"kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx\") pod \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") "
Oct 30 18:11:36.453422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453325    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453357    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453392    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453461    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453503    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453536    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453606    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453622    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453631    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert\") pod \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") "
Oct 30 18:11:36.453774 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453740    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs\") pod \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\" (UID: \"9f9faf67-3b4f-454a-a7cc-6d95fea1997b\") "
Oct 30 18:11:36.454197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453839    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:36.454197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.453879    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data\") pod \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\" (UID: \"cab480c1-9f5e-4933-968f-7b3e9feb0b75\") "
Oct 30 18:11:36.454197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.454186    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.454371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.454202    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.454371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.454216    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.455734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.454792    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:36.455734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.455684    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:36.456835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.456790    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert" pod "9f9faf67-3b4f-454a-a7cc-6d95fea1997b" (UID: "9f9faf67-3b4f-454a-a7cc-6d95fea1997b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:11:36.457173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.457144    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx" pod "9f9faf67-3b4f-454a-a7cc-6d95fea1997b" (UID: "9f9faf67-3b4f-454a-a7cc-6d95fea1997b"). InnerVolumeSpecName "kube-api-access-rcjpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:11:36.459233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.459207    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:11:36.459395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.459374    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:11:36.459471 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.459388    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:11:36.459535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.459487    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs" pod "9f9faf67-3b4f-454a-a7cc-6d95fea1997b" (UID: "9f9faf67-3b4f-454a-a7cc-6d95fea1997b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:11:36.461302 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.461273    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs" pod "cab480c1-9f5e-4933-968f-7b3e9feb0b75" (UID: "cab480c1-9f5e-4933-968f-7b3e9feb0b75"). InnerVolumeSpecName "kube-api-access-dwdqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:11:36.555516 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555480    5534 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-webhook-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555516 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555516    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555531    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555545    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjpx\" (UniqueName: \"kubernetes.io/projected/9f9faf67-3b4f-454a-a7cc-6d95fea1997b-kube-api-access-rcjpx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555560    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555572    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555589    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwdqs\" (UniqueName: \"kubernetes.io/projected/cab480c1-9f5e-4933-968f-7b3e9feb0b75-kube-api-access-dwdqs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555602    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.555801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.555615    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab480c1-9f5e-4933-968f-7b3e9feb0b75-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:11:36.742911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.742875    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"]
Oct 30 18:11:36.746741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.746721    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-tsc4p"]
Oct 30 18:11:36.779802 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.779772    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bb675f547-6kmcm"]
Oct 30 18:11:36.783390 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:36.783285    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bb675f547-6kmcm"]
Oct 30 18:11:37.671851 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.671811    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bb675f547-hpc2m"]
Oct 30 18:11:37.672348 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672323    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9f9faf67-3b4f-454a-a7cc-6d95fea1997b" containerName="manager"
Oct 30 18:11:37.672348 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672342    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9faf67-3b4f-454a-a7cc-6d95fea1997b" containerName="manager"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672363    5534 cpu_manager.go:414] "containerMap: removing container" podUID="fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" containerName="revoke-keys"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672371    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" containerName="revoke-keys"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672404    5534 cpu_manager.go:414] "containerMap: removing container" podUID="01deb567-0ca4-47f9-8118-b26b478a4226" containerName="revoke-keys"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672413    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01deb567-0ca4-47f9-8118-b26b478a4226" containerName="revoke-keys"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672426    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672433    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672444    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main"
Oct 30 18:11:37.672492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672452    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main"
Oct 30 18:11:37.672952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672551    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9faf67-3b4f-454a-a7cc-6d95fea1997b" containerName="manager"
Oct 30 18:11:37.672952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672563    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" containerName="revoke-keys"
Oct 30 18:11:37.672952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672578    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="01deb567-0ca4-47f9-8118-b26b478a4226" containerName="revoke-keys"
Oct 30 18:11:37.672952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672587    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" containerName="istio-proxy"
Oct 30 18:11:37.672952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.672595    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c768139-2889-4936-b9bd-34fd5d0718fd" containerName="main"
Oct 30 18:11:37.677482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.677461    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.691995 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.691972    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 18:11:37.704057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.704026    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 18:11:37.704057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.704030    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kz7qf\""
Oct 30 18:11:37.704214 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.704096    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 18:11:37.706066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.706044    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb675f547-hpc2m"]
Oct 30 18:11:37.767992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.767951    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.768208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.768017    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g827j\" (UniqueName: \"kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.768208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.768145    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.869526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.869489    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.869766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.869552    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.869766 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.869599    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g827j\" (UniqueName: \"kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.872676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.872628    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.872797 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.872636    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.892532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.892496    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g827j\" (UniqueName: \"kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j\") pod \"maas-controller-5bb675f547-hpc2m\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") " pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:37.990308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:37.990218    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:38.299622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.299576    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb675f547-hpc2m"]
Oct 30 18:11:38.304508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.304489    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:11:38.307037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.307002    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:38.368926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.368894    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9faf67-3b4f-454a-a7cc-6d95fea1997b" path="/var/lib/kubelet/pods/9f9faf67-3b4f-454a-a7cc-6d95fea1997b/volumes"
Oct 30 18:11:38.369404 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.369384    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab480c1-9f5e-4933-968f-7b3e9feb0b75" path="/var/lib/kubelet/pods/cab480c1-9f5e-4933-968f-7b3e9feb0b75/volumes"
Oct 30 18:11:38.415227 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.415195    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-hpc2m" event={"ID":"9eb0d022-98c3-456a-b95c-afd14cc54ded","Type":"ContainerStarted","Data":"3da1bb8f0ec6d832df6aab064a0e0a8e749d0248267ef71ba6e04eb0ce5b4873"}
Oct 30 18:11:38.430349 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:38.430322    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:39.384202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:39.384164    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:39.421994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:39.421960    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-hpc2m" event={"ID":"9eb0d022-98c3-456a-b95c-afd14cc54ded","Type":"ContainerStarted","Data":"95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9"}
Oct 30 18:11:39.422162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:39.422063    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:39.463157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:39.463098    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5bb675f547-hpc2m" podStartSLOduration=1.929521198 podStartE2EDuration="2.463081961s" totalImagesPullingTime="533.560763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:39.462582585 +0000 UTC m=+3505.224626401" watchObservedRunningTime="2026-10-30 18:11:39.463081961 +0000 UTC m=+3505.225125791"
Oct 30 18:11:45.101918 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:45.101876    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:11:45.243823 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:11:45.243787    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:11:50.202305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.202266    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"]
Oct 30 18:11:50.205968 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.205946    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.237356 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.237320    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"]
Oct 30 18:11:50.383587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383557    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.383796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383652    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bntb\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.383796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383709    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.383796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383753    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.383924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383842    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.383924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383865    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.383924 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383888    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.384047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383947    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.384047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.383978    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.429462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.429430    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:11:50.485474 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485380    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485481    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485548    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485583    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485617    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485702    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485734    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485771    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.485942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485797    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bntb\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.486178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.485968    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.486178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.486003    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.486178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.486017    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.486540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.486520    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.486868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.486796    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.488487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.488465    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.489150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.489128    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.508931 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.508896    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.509081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.508976    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bntb\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb\") pod \"maas-default-gateway-openshift-default-7df854bf67-92jb6\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.518760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.518733    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:50.837588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.837549    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"]
Oct 30 18:11:50.839914 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:11:50.839890    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab59453_7b05_46b1_b731_cdf6fb511b38.slice/crio-3f2ef714cb2650cc04e9d59e16067da8399e3b9b03004d2da0ee02e76781e607 WatchSource:0}: Error finding container 3f2ef714cb2650cc04e9d59e16067da8399e3b9b03004d2da0ee02e76781e607: Status 404 returned error can't find the container with id 3f2ef714cb2650cc04e9d59e16067da8399e3b9b03004d2da0ee02e76781e607
Oct 30 18:11:50.842041 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.842013    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:11:50.842113 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.842074    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:11:50.842113 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:50.842106    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:11:51.491080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:51.491013    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" event={"ID":"cab59453-7b05-46b1-b731-cdf6fb511b38","Type":"ContainerStarted","Data":"4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7"}
Oct 30 18:11:51.491480 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:51.491105    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" event={"ID":"cab59453-7b05-46b1-b731-cdf6fb511b38","Type":"ContainerStarted","Data":"3f2ef714cb2650cc04e9d59e16067da8399e3b9b03004d2da0ee02e76781e607"}
Oct 30 18:11:51.519778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:51.519752    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:51.521466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:51.521440    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:51.521588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:51.521486    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:51.550000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:51.549953    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podStartSLOduration=1.54994143 podStartE2EDuration="1.54994143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:51.546069092 +0000 UTC m=+3517.308112917" watchObservedRunningTime="2026-10-30 18:11:51.54994143 +0000 UTC m=+3517.311985243"
Oct 30 18:11:52.519706 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:52.519677    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:52.520095 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:52.519719    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:53.519810 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:53.519777    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:53.522127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:53.519826    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:54.026578 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:54.026547    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"]
Oct 30 18:11:54.038233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:54.038205    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dccee96-k8b7s"]
Oct 30 18:11:54.380835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:54.380796    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b072561-7419-4119-a650-202284e75d91" path="/var/lib/kubelet/pods/9b072561-7419-4119-a650-202284e75d91/volumes"
Oct 30 18:11:54.520017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:54.519937    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:54.520017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:54.519990    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:55.520130 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:55.520102    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:55.520554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:55.520153    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:56.519528 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:56.519494    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:56.519730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:56.519552    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:57.519944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:57.519906    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused" start-of-body=
Oct 30 18:11:57.520489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:57.519965    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": dial tcp 10.133.1.129:15021: connect: connection refused"
Oct 30 18:11:58.425564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.425526    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"]
Oct 30 18:11:58.429100 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.429077    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.460949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.460921    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:11:58.461114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.460955    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:11:58.461114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.461029    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:11:58.461114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.461060    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:11:58.463380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.463353    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"]
Oct 30 18:11:58.561464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.561425    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.562039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.561489    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.562039 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.561579    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6v9\" (UniqueName: \"kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.662945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.662900    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6v9\" (UniqueName: \"kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.663726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.663697    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.664519 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.664501    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.665481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.665347    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.677910 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.677828    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.686297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.686260    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6v9\" (UniqueName: \"kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9\") pod \"maas-api-revoke-sub-85237e45-xczmv\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:58.744114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:58.744069    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:11:59.062854 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.062758    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"]
Oct 30 18:11:59.246488 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.246456    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:59.247165 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.247137    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:59.247999 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.247981    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:11:59.537821 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.537783    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv" event={"ID":"1528f365-305b-45ca-bc80-30aaea66cd87","Type":"ContainerStarted","Data":"d7b1d66e3c8c7e9ad0bc5a48e10d3e226b696a89a5eeddcd4d24610e756c218d"}
Oct 30 18:11:59.538011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.537846    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv" event={"ID":"1528f365-305b-45ca-bc80-30aaea66cd87","Type":"ContainerStarted","Data":"ce9d6ebba1426f3f5c557214a3651ce2efc845a28b18cb3781e0ba993d1801c2"}
Oct 30 18:11:59.576276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:11:59.576186    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv" podStartSLOduration=1.576167431 podStartE2EDuration="1.576167431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:59.57577442 +0000 UTC m=+3525.337818234" watchObservedRunningTime="2026-10-30 18:11:59.576167431 +0000 UTC m=+3525.338211243"
Oct 30 18:12:00.546729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:00.546687    5534 generic.go:356] "Generic (PLEG): container finished" podID="1528f365-305b-45ca-bc80-30aaea66cd87" containerID="d7b1d66e3c8c7e9ad0bc5a48e10d3e226b696a89a5eeddcd4d24610e756c218d" exitCode=0
Oct 30 18:12:00.546957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:00.546765    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv" event={"ID":"1528f365-305b-45ca-bc80-30aaea66cd87","Type":"ContainerDied","Data":"d7b1d66e3c8c7e9ad0bc5a48e10d3e226b696a89a5eeddcd4d24610e756c218d"}
Oct 30 18:12:01.729088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.729057    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:12:01.809438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.809305    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9\" (UniqueName: \"kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9\") pod \"1528f365-305b-45ca-bc80-30aaea66cd87\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") "
Oct 30 18:12:01.809438 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.809405    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca\") pod \"1528f365-305b-45ca-bc80-30aaea66cd87\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") "
Oct 30 18:12:01.809776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.809485    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca\") pod \"1528f365-305b-45ca-bc80-30aaea66cd87\" (UID: \"1528f365-305b-45ca-bc80-30aaea66cd87\") "
Oct 30 18:12:01.810187 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.810144    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca" pod "1528f365-305b-45ca-bc80-30aaea66cd87" (UID: "1528f365-305b-45ca-bc80-30aaea66cd87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:12:01.812330 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.812288    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9" pod "1528f365-305b-45ca-bc80-30aaea66cd87" (UID: "1528f365-305b-45ca-bc80-30aaea66cd87"). InnerVolumeSpecName "kube-api-access-cj6v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:12:01.812614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.812574    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca" pod "1528f365-305b-45ca-bc80-30aaea66cd87" (UID: "1528f365-305b-45ca-bc80-30aaea66cd87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:12:01.911133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.911078    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:12:01.911133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.911131    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1528f365-305b-45ca-bc80-30aaea66cd87-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:12:01.911375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:01.911147    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6v9\" (UniqueName: \"kubernetes.io/projected/1528f365-305b-45ca-bc80-30aaea66cd87-kube-api-access-cj6v9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:12:02.560056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:02.560025    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"
Oct 30 18:12:02.560056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:02.560041    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv" event={"ID":"1528f365-305b-45ca-bc80-30aaea66cd87","Type":"ContainerDied","Data":"ce9d6ebba1426f3f5c557214a3651ce2efc845a28b18cb3781e0ba993d1801c2"}
Oct 30 18:12:02.560340 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:02.560088    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9d6ebba1426f3f5c557214a3651ce2efc845a28b18cb3781e0ba993d1801c2"
Oct 30 18:12:04.754675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:04.754601    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:07.044206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:07.044171    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"]
Oct 30 18:12:07.056055 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:07.055928    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f00c798-shsmp"]
Oct 30 18:12:08.367609 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:08.367575    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284aac15-7f8e-43d9-8daf-4c6ae33e0a1b" path="/var/lib/kubelet/pods/284aac15-7f8e-43d9-8daf-4c6ae33e0a1b/volumes"
Oct 30 18:12:15.243480 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:12:15.243438    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:12:20.011929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.011896    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:20.177159 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.177124    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bb675f547-hpc2m"]
Oct 30 18:12:20.177353 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.177314    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5bb675f547-hpc2m" podUID="9eb0d022-98c3-456a-b95c-afd14cc54ded" containerName="manager" containerID="cri-o://95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9" gracePeriod=30
Oct 30 18:12:20.426085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.426061    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:12:20.586037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.585955    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs\") pod \"9eb0d022-98c3-456a-b95c-afd14cc54ded\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") "
Oct 30 18:12:20.586037 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.586001    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j\" (UniqueName: \"kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j\") pod \"9eb0d022-98c3-456a-b95c-afd14cc54ded\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") "
Oct 30 18:12:20.586308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.586110    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert\") pod \"9eb0d022-98c3-456a-b95c-afd14cc54ded\" (UID: \"9eb0d022-98c3-456a-b95c-afd14cc54ded\") "
Oct 30 18:12:20.588190 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.588160    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j" pod "9eb0d022-98c3-456a-b95c-afd14cc54ded" (UID: "9eb0d022-98c3-456a-b95c-afd14cc54ded"). InnerVolumeSpecName "kube-api-access-g827j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:12:20.588308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.588252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert" pod "9eb0d022-98c3-456a-b95c-afd14cc54ded" (UID: "9eb0d022-98c3-456a-b95c-afd14cc54ded"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:12:20.588308 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.588266    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs" pod "9eb0d022-98c3-456a-b95c-afd14cc54ded" (UID: "9eb0d022-98c3-456a-b95c-afd14cc54ded"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:12:20.656331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.656300    5534 generic.go:356] "Generic (PLEG): container finished" podID="9eb0d022-98c3-456a-b95c-afd14cc54ded" containerID="95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9" exitCode=0
Oct 30 18:12:20.656490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.656370    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-hpc2m"
Oct 30 18:12:20.656490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.656378    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-hpc2m" event={"ID":"9eb0d022-98c3-456a-b95c-afd14cc54ded","Type":"ContainerDied","Data":"95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9"}
Oct 30 18:12:20.656490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.656439    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-hpc2m" event={"ID":"9eb0d022-98c3-456a-b95c-afd14cc54ded","Type":"ContainerDied","Data":"3da1bb8f0ec6d832df6aab064a0e0a8e749d0248267ef71ba6e04eb0ce5b4873"}
Oct 30 18:12:20.656490 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.656461    5534 scope.go:122] "RemoveContainer" containerID="95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9"
Oct 30 18:12:20.666086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.666068    5534 scope.go:122] "RemoveContainer" containerID="95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9"
Oct 30 18:12:20.666323 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:12:20.666307    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9\": container with ID starting with 95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9 not found: ID does not exist" containerID="95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9"
Oct 30 18:12:20.666379 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.666333    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9"} err="failed to get container status \"95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9\": rpc error: code = NotFound desc = could not find container \"95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9\": container with ID starting with 95a8f37cd1b817f830b64223e6438a2d37f69f7524d12f140fdea185bf6345a9 not found: ID does not exist"
Oct 30 18:12:20.686882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.686860    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:12:20.686997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.686885    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g827j\" (UniqueName: \"kubernetes.io/projected/9eb0d022-98c3-456a-b95c-afd14cc54ded-kube-api-access-g827j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:12:20.686997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.686899    5534 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eb0d022-98c3-456a-b95c-afd14cc54ded-webhook-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:12:20.704741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.704717    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bb675f547-hpc2m"]
Oct 30 18:12:20.708431 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:20.708403    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bb675f547-hpc2m"]
Oct 30 18:12:21.164973 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.164943    5534 scope.go:122] "RemoveContainer" containerID="40f4d7cdd6dbe0e3a0476068e4590c67fbaab52c9ba1edddf7d7678b0d022859"
Oct 30 18:12:21.174171 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.174147    5534 scope.go:122] "RemoveContainer" containerID="9a12576e57aedcae276536344e37427d57769cb19aa81923443d5e90bb64aa68"
Oct 30 18:12:21.182335 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.182133    5534 scope.go:122] "RemoveContainer" containerID="d44036016adbb01bab421b7d57a9a372f2bdafcd6741394ebfc4b0d928b463ae"
Oct 30 18:12:21.190036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.190013    5534 scope.go:122] "RemoveContainer" containerID="191a72a4b6ac2fcfe9b78b87bbbefff5673f77e0a194b7e298fdff9e22911fe8"
Oct 30 18:12:21.198046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.198021    5534 scope.go:122] "RemoveContainer" containerID="9e73098842bff95a3812431ac82b44d0463054d1c62e21e73d0a276abafb5a12"
Oct 30 18:12:21.206214 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.206192    5534 scope.go:122] "RemoveContainer" containerID="f381e03c0525d058aba91b7e7d4a3c7bff02ad04eadd47110fa4e1a8f298b1cb"
Oct 30 18:12:21.214538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.214518    5534 scope.go:122] "RemoveContainer" containerID="9acc881e506c4a449db4643a1fe6e5f1fc55c1a27b52d9f7a5a3d0575416ee48"
Oct 30 18:12:21.225147 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.225126    5534 scope.go:122] "RemoveContainer" containerID="2ef1fd9934355367f514172906ef85972e034eac6e9b172cbb4247cd0b0d3116"
Oct 30 18:12:21.233813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.233790    5534 scope.go:122] "RemoveContainer" containerID="211cc7dd7c85caa1348996cfe547c835e38ae4df5bc8e9d8eda3da97f931cb06"
Oct 30 18:12:21.799716 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.799680    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:21.917367 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917336    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bb675f547-mb27n"]
Oct 30 18:12:21.917729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917716    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1528f365-305b-45ca-bc80-30aaea66cd87" containerName="revoke-keys"
Oct 30 18:12:21.917785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917730    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1528f365-305b-45ca-bc80-30aaea66cd87" containerName="revoke-keys"
Oct 30 18:12:21.917785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917753    5534 cpu_manager.go:414] "containerMap: removing container" podUID="9eb0d022-98c3-456a-b95c-afd14cc54ded" containerName="manager"
Oct 30 18:12:21.917785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917758    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb0d022-98c3-456a-b95c-afd14cc54ded" containerName="manager"
Oct 30 18:12:21.917926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917812    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1528f365-305b-45ca-bc80-30aaea66cd87" containerName="revoke-keys"
Oct 30 18:12:21.917926 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.917826    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb0d022-98c3-456a-b95c-afd14cc54ded" containerName="manager"
Oct 30 18:12:21.923005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.922984    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:21.948783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.948750    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kz7qf\""
Oct 30 18:12:21.948783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.948760    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 18:12:21.948783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.948760    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 18:12:21.949094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.948811    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 18:12:21.950461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.950441    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb675f547-mb27n"]
Oct 30 18:12:21.998027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.997998    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eed760b-e4dc-4f18-aa57-a58ff18df081-metrics-certs\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:21.998027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.998040    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jb5\" (UniqueName: \"kubernetes.io/projected/4eed760b-e4dc-4f18-aa57-a58ff18df081-kube-api-access-t6jb5\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:21.998243 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:21.998073    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4eed760b-e4dc-4f18-aa57-a58ff18df081-webhook-cert\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.099419 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.099391    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eed760b-e4dc-4f18-aa57-a58ff18df081-metrics-certs\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.099582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.099432    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jb5\" (UniqueName: \"kubernetes.io/projected/4eed760b-e4dc-4f18-aa57-a58ff18df081-kube-api-access-t6jb5\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.099582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.099547    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4eed760b-e4dc-4f18-aa57-a58ff18df081-webhook-cert\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.102097 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.102073    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4eed760b-e4dc-4f18-aa57-a58ff18df081-webhook-cert\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.102221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.102122    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eed760b-e4dc-4f18-aa57-a58ff18df081-metrics-certs\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.122363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.122337    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jb5\" (UniqueName: \"kubernetes.io/projected/4eed760b-e4dc-4f18-aa57-a58ff18df081-kube-api-access-t6jb5\") pod \"maas-controller-5bb675f547-mb27n\" (UID: \"4eed760b-e4dc-4f18-aa57-a58ff18df081\") " pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.235607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.235572    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:22.368463 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.368385    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb0d022-98c3-456a-b95c-afd14cc54ded" path="/var/lib/kubelet/pods/9eb0d022-98c3-456a-b95c-afd14cc54ded/volumes"
Oct 30 18:12:22.534022 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.533996    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bb675f547-mb27n"]
Oct 30 18:12:22.536616 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:12:22.536594    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eed760b_e4dc_4f18_aa57_a58ff18df081.slice/crio-93edda93fa6609a4cb73dfa24837b8d44ea0f58c6c302e4293b2257dd75d027d WatchSource:0}: Error finding container 93edda93fa6609a4cb73dfa24837b8d44ea0f58c6c302e4293b2257dd75d027d: Status 404 returned error can't find the container with id 93edda93fa6609a4cb73dfa24837b8d44ea0f58c6c302e4293b2257dd75d027d
Oct 30 18:12:22.667802 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:22.667724    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-mb27n" event={"ID":"4eed760b-e4dc-4f18-aa57-a58ff18df081","Type":"ContainerStarted","Data":"93edda93fa6609a4cb73dfa24837b8d44ea0f58c6c302e4293b2257dd75d027d"}
Oct 30 18:12:23.675236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:23.675197    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bb675f547-mb27n" event={"ID":"4eed760b-e4dc-4f18-aa57-a58ff18df081","Type":"ContainerStarted","Data":"02b262cbd79cef7b2d951cf27e3e5f760f51740eabd46b83dc33157778260e39"}
Oct 30 18:12:23.675744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:23.675352    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:23.716828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:23.716771    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5bb675f547-mb27n" podStartSLOduration=2.349054979 podStartE2EDuration="2.716756386s" totalImagesPullingTime="367.701407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:23.715971021 +0000 UTC m=+3549.478014857" watchObservedRunningTime="2026-10-30 18:12:23.716756386 +0000 UTC m=+3549.478800210"
Oct 30 18:12:24.486185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:24.486150    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:24.603778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:24.603740    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:33.504618 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:33.504570    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:34.683376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:34.683345    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bb675f547-mb27n"
Oct 30 18:12:45.243738 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:12:45.243701    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:12:58.675034 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.674998    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:58.776305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.776262    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"]
Oct 30 18:12:58.780364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.780338    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.794616 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.794586    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:12:58.806371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.806342    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:12:58.806668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.806628    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:12:58.806833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.806800    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:12:58.807925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.807900    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"]
Oct 30 18:12:58.823350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.823288    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:12:58.858377 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.858344    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlj6p\" (UniqueName: \"kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.858514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.858382    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.858514 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.858498    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.959942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.959881    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.960059 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.959977    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlj6p\" (UniqueName: \"kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.960059 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.960006    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.960665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.960629    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.962278 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.962262    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:58.982507 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:58.982488    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlj6p\" (UniqueName: \"kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p\") pod \"maas-api-revoke-sub-c1e271a9-2r5xh\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:59.094360 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:59.094331    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:12:59.390549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:59.390529    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"]
Oct 30 18:12:59.392632 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:12:59.392606    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5102cfd_2685_4cb8_ba0e_4faae7dea198.slice/crio-32e397dea6ff997d709e60c7afe756dcb52ea1f2eda91e1ba6361735df266c44 WatchSource:0}: Error finding container 32e397dea6ff997d709e60c7afe756dcb52ea1f2eda91e1ba6361735df266c44: Status 404 returned error can't find the container with id 32e397dea6ff997d709e60c7afe756dcb52ea1f2eda91e1ba6361735df266c44
Oct 30 18:12:59.863990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:59.863951    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh" event={"ID":"b5102cfd-2685-4cb8-ba0e-4faae7dea198","Type":"ContainerStarted","Data":"1af86b7c0fcbbc277f083ca0d0310a95fbfe3a23fcedfc366bea6dd6d9c4cfa5"}
Oct 30 18:12:59.864398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:59.864054    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh" event={"ID":"b5102cfd-2685-4cb8-ba0e-4faae7dea198","Type":"ContainerStarted","Data":"32e397dea6ff997d709e60c7afe756dcb52ea1f2eda91e1ba6361735df266c44"}
Oct 30 18:12:59.904453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:12:59.904392    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh" podStartSLOduration=1.9043741870000002 podStartE2EDuration="1.904374187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:59.902571268 +0000 UTC m=+3585.664615082" watchObservedRunningTime="2026-10-30 18:12:59.904374187 +0000 UTC m=+3585.666418002"
Oct 30 18:13:00.870659 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:00.870604    5534 generic.go:356] "Generic (PLEG): container finished" podID="b5102cfd-2685-4cb8-ba0e-4faae7dea198" containerID="1af86b7c0fcbbc277f083ca0d0310a95fbfe3a23fcedfc366bea6dd6d9c4cfa5" exitCode=0
Oct 30 18:13:00.871064 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:00.870693    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh" event={"ID":"b5102cfd-2685-4cb8-ba0e-4faae7dea198","Type":"ContainerDied","Data":"1af86b7c0fcbbc277f083ca0d0310a95fbfe3a23fcedfc366bea6dd6d9c4cfa5"}
Oct 30 18:13:02.027150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.027124    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:13:02.089429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.089400    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca\") pod \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") "
Oct 30 18:13:02.089673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.089457    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca\") pod \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") "
Oct 30 18:13:02.089673 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.089532    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p\" (UniqueName: \"kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p\") pod \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\" (UID: \"b5102cfd-2685-4cb8-ba0e-4faae7dea198\") "
Oct 30 18:13:02.089843 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.089818    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca" pod "b5102cfd-2685-4cb8-ba0e-4faae7dea198" (UID: "b5102cfd-2685-4cb8-ba0e-4faae7dea198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:13:02.089987 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.089933    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:02.091750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.091725    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca" pod "b5102cfd-2685-4cb8-ba0e-4faae7dea198" (UID: "b5102cfd-2685-4cb8-ba0e-4faae7dea198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:13:02.091839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.091791    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p" pod "b5102cfd-2685-4cb8-ba0e-4faae7dea198" (UID: "b5102cfd-2685-4cb8-ba0e-4faae7dea198"). InnerVolumeSpecName "kube-api-access-qlj6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:13:02.190838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.190733    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5102cfd-2685-4cb8-ba0e-4faae7dea198-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:02.190838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.190780    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlj6p\" (UniqueName: \"kubernetes.io/projected/b5102cfd-2685-4cb8-ba0e-4faae7dea198-kube-api-access-qlj6p\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:02.881791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.881745    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh" event={"ID":"b5102cfd-2685-4cb8-ba0e-4faae7dea198","Type":"ContainerDied","Data":"32e397dea6ff997d709e60c7afe756dcb52ea1f2eda91e1ba6361735df266c44"}
Oct 30 18:13:02.881791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.881788    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"
Oct 30 18:13:02.882024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:02.881804    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32e397dea6ff997d709e60c7afe756dcb52ea1f2eda91e1ba6361735df266c44"
Oct 30 18:13:04.870764 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:04.870719    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:05.408202 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.408172    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"]
Oct 30 18:13:05.408584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.408568    5534 cpu_manager.go:414] "containerMap: removing container" podUID="b5102cfd-2685-4cb8-ba0e-4faae7dea198" containerName="revoke-keys"
Oct 30 18:13:05.408674 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.408585    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5102cfd-2685-4cb8-ba0e-4faae7dea198" containerName="revoke-keys"
Oct 30 18:13:05.408722 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.408679    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5102cfd-2685-4cb8-ba0e-4faae7dea198" containerName="revoke-keys"
Oct 30 18:13:05.412089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.412064    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.443170 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.443143    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"]
Oct 30 18:13:05.445030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.445007    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"]
Oct 30 18:13:05.449201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.449183    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.478279 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.478250    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"]
Oct 30 18:13:05.522730 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522702    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.522857 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522739    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.522857 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522809    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.522937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522868    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.522937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522890    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.522937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522906    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.522978    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523014    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523059    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523078    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.523118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523107    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523125    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.523216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523147    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.523216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523180    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523217    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzcs\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.523312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523262    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.523312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523291    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l265v\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.523312 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.523310    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624271    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624318    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624337    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624357    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624383    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624415    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624442    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624467    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624489    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624511    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624534    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624569    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624598    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624636    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzcs\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624703    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624746    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l265v\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624775    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624811    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.624837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624827    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.625531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.624849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.625531 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.625381    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.625606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.625560    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.625724 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.625702    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.625868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.625849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.626070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.626042    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.626188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.626090    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.626257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.626238    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.626328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.626311    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.627075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.627042    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.627232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.627213    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.627371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.627353    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.628000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.627981    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.647338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.647317    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.647689 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.647674    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.647741 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.647708    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzcs\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8ttz7\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.647830 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.647817    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l265v\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v\") pod \"maas-default-gateway-openshift-default-7df854bf67-j4pdf\" (UID: \"5757d303-8049-4309-8652-001808326e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:05.727271 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.727160    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:05.763445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:05.763408    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:06.044718 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.044690    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"]
Oct 30 18:13:06.046820 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:13:06.046793    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3be3002_8b68_42c9_8a9a_74e2774e277b.slice/crio-3ebf85865ebdccca22d668ea3b2c143a65fadf22c584b0e81dd6c5a1947ad987 WatchSource:0}: Error finding container 3ebf85865ebdccca22d668ea3b2c143a65fadf22c584b0e81dd6c5a1947ad987: Status 404 returned error can't find the container with id 3ebf85865ebdccca22d668ea3b2c143a65fadf22c584b0e81dd6c5a1947ad987
Oct 30 18:13:06.049072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.049039    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:13:06.049176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.049104    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:13:06.049176 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.049131    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:13:06.091280 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.091258    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"]
Oct 30 18:13:06.092105 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:13:06.092080    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5757d303_8049_4309_8652_001808326e91.slice/crio-d8ae411de7d659de669d80c4d867bb86471db7f09668021a8f856cda730ddace WatchSource:0}: Error finding container d8ae411de7d659de669d80c4d867bb86471db7f09668021a8f856cda730ddace: Status 404 returned error can't find the container with id d8ae411de7d659de669d80c4d867bb86471db7f09668021a8f856cda730ddace
Oct 30 18:13:06.094348 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.094321    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:13:06.094434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.094379    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:13:06.094434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.094407    5534 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:13:06.912141 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.912099    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" event={"ID":"5757d303-8049-4309-8652-001808326e91","Type":"ContainerStarted","Data":"7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69"}
Oct 30 18:13:06.912350 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.912249    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" event={"ID":"5757d303-8049-4309-8652-001808326e91","Type":"ContainerStarted","Data":"d8ae411de7d659de669d80c4d867bb86471db7f09668021a8f856cda730ddace"}
Oct 30 18:13:06.913920 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.913899    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" event={"ID":"f3be3002-8b68-42c9-8a9a-74e2774e277b","Type":"ContainerStarted","Data":"7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084"}
Oct 30 18:13:06.914043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.913940    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" event={"ID":"f3be3002-8b68-42c9-8a9a-74e2774e277b","Type":"ContainerStarted","Data":"3ebf85865ebdccca22d668ea3b2c143a65fadf22c584b0e81dd6c5a1947ad987"}
Oct 30 18:13:06.993671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:06.991529    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podStartSLOduration=1.9915083359999999 podStartE2EDuration="1.991508336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:06.974348757 +0000 UTC m=+3592.736392571" watchObservedRunningTime="2026-10-30 18:13:06.991508336 +0000 UTC m=+3592.753552151"
Oct 30 18:13:07.024287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.024226    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podStartSLOduration=2.024211123 podStartE2EDuration="2.024211123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:07.022846506 +0000 UTC m=+3592.784890331" watchObservedRunningTime="2026-10-30 18:13:07.024211123 +0000 UTC m=+3592.786255003"
Oct 30 18:13:07.727367 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.727337    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:07.729311 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.729287    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused" start-of-body=
Oct 30 18:13:07.729440 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.729332    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused"
Oct 30 18:13:07.767439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.767404    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body=
Oct 30 18:13:07.767767 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.767725    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused"
Oct 30 18:13:07.768294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:07.768275    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:08.727758 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:08.727718    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused" start-of-body=
Oct 30 18:13:08.728288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:08.727784    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused"
Oct 30 18:13:08.765500 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:08.765455    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body=
Oct 30 18:13:08.765837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:08.765791    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused"
Oct 30 18:13:09.727542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:09.727504    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused" start-of-body=
Oct 30 18:13:09.727785 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:09.727561    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused"
Oct 30 18:13:09.770671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:09.765393    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body=
Oct 30 18:13:09.770671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:09.765468    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused"
Oct 30 18:13:10.728371 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:10.728337    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused" start-of-body=
Oct 30 18:13:10.728906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:10.728874    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused"
Oct 30 18:13:10.767668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:10.767014    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body=
Oct 30 18:13:10.767668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:10.767073    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused"
Oct 30 18:13:11.728001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:11.727967    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused" start-of-body=
Oct 30 18:13:11.728181 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:11.728016    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused"
Oct 30 18:13:11.770636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:11.770045    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body=
Oct 30 18:13:11.770636 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:11.770110    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused"
Oct 30 18:13:12.727987 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:12.727946    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused" start-of-body=
Oct 30 18:13:12.728173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:12.728015    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": dial tcp 10.133.1.133:15021: connect: connection refused"
Oct 30 18:13:12.764701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:12.764667    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused" start-of-body=
Oct 30 18:13:12.764859 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:12.764730    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": dial tcp 10.133.1.134:15021: connect: connection refused"
Oct 30 18:13:14.729428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:14.729377    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:13:14.729892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:14.729477    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:13:14.764727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:14.764686    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:13:14.764925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:14.764752    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:13:15.246292 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:13:15.246251    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:13:15.770823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:15.770784    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:13:15.771465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:15.771421    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:13:15.805807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:15.805773    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:13:15.805945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:15.805839    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:13:16.772508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.772465    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:13:16.773080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.772530    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:13:16.846806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.846777    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:13:16.846965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.846828    5534 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:13:16.872745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.872707    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:16.873172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.873148    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:16.874247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:16.874224    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:13:17.090054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:17.090029    5534 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:17.090467 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:17.090435    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:17.091131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:17.091114    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:13:20.123403 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:20.123367    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:20.260066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:20.260031    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:21.299892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:21.299863    5534 scope.go:122] "RemoveContainer" containerID="fdd208a076c1352c2addd7ceec51ea27c157a24cd05b92d97d7a29ebf739be7e"
Oct 30 18:13:21.308909 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:21.308886    5534 scope.go:122] "RemoveContainer" containerID="a35320ea7d8f194f65142e9ad038ab1bd41cf45be412cac80cc4506e1ff66a91"
Oct 30 18:13:21.317834 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:21.317804    5534 scope.go:122] "RemoveContainer" containerID="08aeef3885eb45d1b857521cc44095dd6c1bf66d747570f80e0068bdb684b843"
Oct 30 18:13:21.327413 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:21.327382    5534 scope.go:122] "RemoveContainer" containerID="149322fb854dfcba965182e09d2e93c92da053e6d20b4fc8471b58eab0f6bf64"
Oct 30 18:13:21.336827 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:21.336801    5534 scope.go:122] "RemoveContainer" containerID="730c4a7f866b192a2100316d3a70e13d68e6db29f137aecc71fb9b2139147393"
Oct 30 18:13:29.827468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.827369    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"]
Oct 30 18:13:29.832255 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.832225    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:29.848676 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.848637    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:13:29.851217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.851185    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"]
Oct 30 18:13:29.860690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.860624    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:13:29.860690 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.860670    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:13:29.860881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.860717    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:13:29.948245 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.948213    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:29.948449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.948348    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:29.948449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:29.948437    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zh6g\" (UniqueName: \"kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.049579 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.049549    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.049831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.049631    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.049831 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.049706    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zh6g\" (UniqueName: \"kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.050290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.050268    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.052215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.052187    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.072631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.072612    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zh6g\" (UniqueName: \"kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g\") pod \"maas-api-revoke-sub-2e1f5b5f-gsblj\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.145249 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.145225    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:30.442473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:30.442395    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"]
Oct 30 18:13:30.445452 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:13:30.445418    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc362a7f4_d59b_49ee_9fdf_3f37cbfafe43.slice/crio-4a2f471dbf602306a890e814e266cc5c0ac3ca7ea718ad4ea4d40e4ab4b2410e WatchSource:0}: Error finding container 4a2f471dbf602306a890e814e266cc5c0ac3ca7ea718ad4ea4d40e4ab4b2410e: Status 404 returned error can't find the container with id 4a2f471dbf602306a890e814e266cc5c0ac3ca7ea718ad4ea4d40e4ab4b2410e
Oct 30 18:13:31.071189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:31.071104    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj" event={"ID":"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43","Type":"ContainerStarted","Data":"1999c8c5b44b6d704b9ddaaebcc8641ea84a83f9500afaecb76583b5c9643e7c"}
Oct 30 18:13:31.071189 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:31.071166    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj" event={"ID":"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43","Type":"ContainerStarted","Data":"4a2f471dbf602306a890e814e266cc5c0ac3ca7ea718ad4ea4d40e4ab4b2410e"}
Oct 30 18:13:31.137445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:31.137386    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj" podStartSLOduration=2.137372199 podStartE2EDuration="2.137372199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:31.136933724 +0000 UTC m=+3616.898977537" watchObservedRunningTime="2026-10-30 18:13:31.137372199 +0000 UTC m=+3616.899416032"
Oct 30 18:13:32.077841 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:32.077804    5534 generic.go:356] "Generic (PLEG): container finished" podID="c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" containerID="1999c8c5b44b6d704b9ddaaebcc8641ea84a83f9500afaecb76583b5c9643e7c" exitCode=0
Oct 30 18:13:32.078248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:32.077940    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj" event={"ID":"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43","Type":"ContainerDied","Data":"1999c8c5b44b6d704b9ddaaebcc8641ea84a83f9500afaecb76583b5c9643e7c"}
Oct 30 18:13:33.216357 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.216333    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:33.281808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.281780    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca\") pod \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") "
Oct 30 18:13:33.281972 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.281882    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g\" (UniqueName: \"kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g\") pod \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") "
Oct 30 18:13:33.281972 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.281919    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca\") pod \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\" (UID: \"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43\") "
Oct 30 18:13:33.282188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.282163    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca" pod "c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" (UID: "c362a7f4-d59b-49ee-9fdf-3f37cbfafe43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:13:33.282328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.282307    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:33.284137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.284117    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca" pod "c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" (UID: "c362a7f4-d59b-49ee-9fdf-3f37cbfafe43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:13:33.284241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.284138    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g" pod "c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" (UID: "c362a7f4-d59b-49ee-9fdf-3f37cbfafe43"). InnerVolumeSpecName "kube-api-access-9zh6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:13:33.383299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.383272    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zh6g\" (UniqueName: \"kubernetes.io/projected/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-kube-api-access-9zh6g\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:33.383299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.383296    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:33.618713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.618665    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:33.935018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:33.934989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:34.088593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:34.088568    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"
Oct 30 18:13:34.088793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:34.088567    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj" event={"ID":"c362a7f4-d59b-49ee-9fdf-3f37cbfafe43","Type":"ContainerDied","Data":"4a2f471dbf602306a890e814e266cc5c0ac3ca7ea718ad4ea4d40e4ab4b2410e"}
Oct 30 18:13:34.088793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:34.088704    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2f471dbf602306a890e814e266cc5c0ac3ca7ea718ad4ea4d40e4ab4b2410e"
Oct 30 18:13:40.995149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:40.995106    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"]
Oct 30 18:13:40.995826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:40.995803    5534 cpu_manager.go:414] "containerMap: removing container" podUID="c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" containerName="revoke-keys"
Oct 30 18:13:40.995930 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:40.995829    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" containerName="revoke-keys"
Oct 30 18:13:40.995987 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:40.995927    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" containerName="revoke-keys"
Oct 30 18:13:41.000528 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.000504    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.015911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.015883    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:13:41.027921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.027890    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:13:41.028161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.027934    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:13:41.028321 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.027983    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:13:41.030024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.030001    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"]
Oct 30 18:13:41.159015 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.158984    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f469j\" (UniqueName: \"kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.159186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.159126    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.159186 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.159172    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.260714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.260609    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.260891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.260719    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.260891 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.260781    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f469j\" (UniqueName: \"kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.261395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.261372    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.263152 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.263129    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.283830 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.283798    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f469j\" (UniqueName: \"kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j\") pod \"maas-api-revoke-sub-ec843018-l887v\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.313760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.313738    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:41.612482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:41.612460    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"]
Oct 30 18:13:41.614670 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:13:41.614625    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e688c04_7d56_4d31_80a0_910b5411c9d5.slice/crio-7c0530564fb05fb334c9c8e770a1a9e2cce962c2ab6b24ce901319599b7797a8 WatchSource:0}: Error finding container 7c0530564fb05fb334c9c8e770a1a9e2cce962c2ab6b24ce901319599b7797a8: Status 404 returned error can't find the container with id 7c0530564fb05fb334c9c8e770a1a9e2cce962c2ab6b24ce901319599b7797a8
Oct 30 18:13:42.130581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:42.130493    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v" event={"ID":"5e688c04-7d56-4d31-80a0-910b5411c9d5","Type":"ContainerStarted","Data":"1d35b9f44baceba6e283b401106b154a5532bfbfe604645d37d7d80f3b544cdb"}
Oct 30 18:13:42.130581 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:42.130553    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v" event={"ID":"5e688c04-7d56-4d31-80a0-910b5411c9d5","Type":"ContainerStarted","Data":"7c0530564fb05fb334c9c8e770a1a9e2cce962c2ab6b24ce901319599b7797a8"}
Oct 30 18:13:42.170090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:42.170042    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v" podStartSLOduration=2.170029354 podStartE2EDuration="2.170029354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:42.168311715 +0000 UTC m=+3627.930355542" watchObservedRunningTime="2026-10-30 18:13:42.170029354 +0000 UTC m=+3627.932073166"
Oct 30 18:13:43.032343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:43.032299    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"]
Oct 30 18:13:43.043841 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:43.043809    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db02afb-9zldm"]
Oct 30 18:13:43.137199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:43.137163    5534 generic.go:356] "Generic (PLEG): container finished" podID="5e688c04-7d56-4d31-80a0-910b5411c9d5" containerID="1d35b9f44baceba6e283b401106b154a5532bfbfe604645d37d7d80f3b544cdb" exitCode=0
Oct 30 18:13:43.137684 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:43.137198    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v" event={"ID":"5e688c04-7d56-4d31-80a0-910b5411c9d5","Type":"ContainerDied","Data":"1d35b9f44baceba6e283b401106b154a5532bfbfe604645d37d7d80f3b544cdb"}
Oct 30 18:13:44.334157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.334129    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:44.369975 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.369944    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96acb75e-3222-4737-97b1-c11821daeda3" path="/var/lib/kubelet/pods/96acb75e-3222-4737-97b1-c11821daeda3/volumes"
Oct 30 18:13:44.496752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.496630    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca\") pod \"5e688c04-7d56-4d31-80a0-910b5411c9d5\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") "
Oct 30 18:13:44.496961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.496750    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j\" (UniqueName: \"kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j\") pod \"5e688c04-7d56-4d31-80a0-910b5411c9d5\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") "
Oct 30 18:13:44.496961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.496789    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca\") pod \"5e688c04-7d56-4d31-80a0-910b5411c9d5\" (UID: \"5e688c04-7d56-4d31-80a0-910b5411c9d5\") "
Oct 30 18:13:44.504221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.501015    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca" pod "5e688c04-7d56-4d31-80a0-910b5411c9d5" (UID: "5e688c04-7d56-4d31-80a0-910b5411c9d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:13:44.516411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.516366    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca" pod "5e688c04-7d56-4d31-80a0-910b5411c9d5" (UID: "5e688c04-7d56-4d31-80a0-910b5411c9d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:13:44.523216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.523177    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j" pod "5e688c04-7d56-4d31-80a0-910b5411c9d5" (UID: "5e688c04-7d56-4d31-80a0-910b5411c9d5"). InnerVolumeSpecName "kube-api-access-f469j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:13:44.599628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.599573    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:44.599628 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.599619    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f469j\" (UniqueName: \"kubernetes.io/projected/5e688c04-7d56-4d31-80a0-910b5411c9d5-kube-api-access-f469j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:44.599951 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.599636    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e688c04-7d56-4d31-80a0-910b5411c9d5-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:13:44.805046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.804952    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:44.926801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:44.926764    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:13:45.153670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:45.151833    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"
Oct 30 18:13:45.153670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:45.151872    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v" event={"ID":"5e688c04-7d56-4d31-80a0-910b5411c9d5","Type":"ContainerDied","Data":"7c0530564fb05fb334c9c8e770a1a9e2cce962c2ab6b24ce901319599b7797a8"}
Oct 30 18:13:45.153670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:13:45.151951    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c0530564fb05fb334c9c8e770a1a9e2cce962c2ab6b24ce901319599b7797a8"
Oct 30 18:13:45.370666 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:13:45.359156    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:14:02.600460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:02.600424    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:14:04.995453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:04.995413    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:14:15.243083 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:14:15.243045    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:14:19.030410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.030379    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"]
Oct 30 18:14:19.030835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.030788    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5e688c04-7d56-4d31-80a0-910b5411c9d5" containerName="revoke-keys"
Oct 30 18:14:19.030835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.030800    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e688c04-7d56-4d31-80a0-910b5411c9d5" containerName="revoke-keys"
Oct 30 18:14:19.030917 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.030881    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e688c04-7d56-4d31-80a0-910b5411c9d5" containerName="revoke-keys"
Oct 30 18:14:19.033788 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.033769    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.059781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.059757    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:14:19.059960 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.059762    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:14:19.060034 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.060009    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:14:19.060095 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.060054    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:14:19.071487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.071450    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"]
Oct 30 18:14:19.140363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.140334    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.140497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.140376    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85nz9\" (UniqueName: \"kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.140572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.140511    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.241423 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.241394    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.241546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.241449    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.241546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.241475    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85nz9\" (UniqueName: \"kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.242040 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.242012    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.243927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.243902    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.264263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.264240    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85nz9\" (UniqueName: \"kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9\") pod \"maas-api-revoke-sub-046e1343-98lhw\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.346957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.346939    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:19.644407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:19.644363    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"]
Oct 30 18:14:19.646800 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:14:19.646762    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97424be_bd9e_4d2a_a380_2315f84ef6c2.slice/crio-9c66e4cb3e1714012370e9ea47dacf172e88142011c9d621b883326b70dcfd27 WatchSource:0}: Error finding container 9c66e4cb3e1714012370e9ea47dacf172e88142011c9d621b883326b70dcfd27: Status 404 returned error can't find the container with id 9c66e4cb3e1714012370e9ea47dacf172e88142011c9d621b883326b70dcfd27
Oct 30 18:14:20.431226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:20.431185    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw" event={"ID":"a97424be-bd9e-4d2a-a380-2315f84ef6c2","Type":"ContainerStarted","Data":"3991bc945751d090feb3ca8d8a252e21bda8789e6ba9931b5163b8f0b22fd740"}
Oct 30 18:14:20.431610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:20.431250    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw" event={"ID":"a97424be-bd9e-4d2a-a380-2315f84ef6c2","Type":"ContainerStarted","Data":"9c66e4cb3e1714012370e9ea47dacf172e88142011c9d621b883326b70dcfd27"}
Oct 30 18:14:20.469922 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:20.469872    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw" podStartSLOduration=1.469857656 podStartE2EDuration="1.469857656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:20.469208507 +0000 UTC m=+3666.231252321" watchObservedRunningTime="2026-10-30 18:14:20.469857656 +0000 UTC m=+3666.231901463"
Oct 30 18:14:21.370261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:21.370232    5534 scope.go:122] "RemoveContainer" containerID="8982bacfa2cbd2f800a2b388ce5e53ba54c398aceecdf05bdd90f506d38a5544"
Oct 30 18:14:21.379316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:21.379292    5534 scope.go:122] "RemoveContainer" containerID="eaa6dea8729c41d5736c3bfad1bacb20bfd44bc22fe1f45e9c527add417da0d0"
Oct 30 18:14:21.437597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:21.437560    5534 generic.go:356] "Generic (PLEG): container finished" podID="a97424be-bd9e-4d2a-a380-2315f84ef6c2" containerID="3991bc945751d090feb3ca8d8a252e21bda8789e6ba9931b5163b8f0b22fd740" exitCode=0
Oct 30 18:14:21.438056 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:21.437673    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw" event={"ID":"a97424be-bd9e-4d2a-a380-2315f84ef6c2","Type":"ContainerDied","Data":"3991bc945751d090feb3ca8d8a252e21bda8789e6ba9931b5163b8f0b22fd740"}
Oct 30 18:14:22.575399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.575368    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:22.671135 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.671101    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca\") pod \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") "
Oct 30 18:14:22.671313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.671221    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9\" (UniqueName: \"kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9\") pod \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") "
Oct 30 18:14:22.671313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.671260    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca\") pod \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\" (UID: \"a97424be-bd9e-4d2a-a380-2315f84ef6c2\") "
Oct 30 18:14:22.671445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.671423    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca" pod "a97424be-bd9e-4d2a-a380-2315f84ef6c2" (UID: "a97424be-bd9e-4d2a-a380-2315f84ef6c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:14:22.673535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.673512    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca" pod "a97424be-bd9e-4d2a-a380-2315f84ef6c2" (UID: "a97424be-bd9e-4d2a-a380-2315f84ef6c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:14:22.673535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.673525    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9" pod "a97424be-bd9e-4d2a-a380-2315f84ef6c2" (UID: "a97424be-bd9e-4d2a-a380-2315f84ef6c2"). InnerVolumeSpecName "kube-api-access-85nz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:14:22.772944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.772858    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:14:22.772944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.772886    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85nz9\" (UniqueName: \"kubernetes.io/projected/a97424be-bd9e-4d2a-a380-2315f84ef6c2-kube-api-access-85nz9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:14:22.772944 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:22.772896    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97424be-bd9e-4d2a-a380-2315f84ef6c2-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:14:23.448781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:23.448749    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"
Oct 30 18:14:23.448985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:23.448750    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw" event={"ID":"a97424be-bd9e-4d2a-a380-2315f84ef6c2","Type":"ContainerDied","Data":"9c66e4cb3e1714012370e9ea47dacf172e88142011c9d621b883326b70dcfd27"}
Oct 30 18:14:23.448985 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:23.448867    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c66e4cb3e1714012370e9ea47dacf172e88142011c9d621b883326b70dcfd27"
Oct 30 18:14:29.442054 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.442020    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"]
Oct 30 18:14:29.442569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.442436    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a97424be-bd9e-4d2a-a380-2315f84ef6c2" containerName="revoke-keys"
Oct 30 18:14:29.442569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.442447    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97424be-bd9e-4d2a-a380-2315f84ef6c2" containerName="revoke-keys"
Oct 30 18:14:29.442569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.442507    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97424be-bd9e-4d2a-a380-2315f84ef6c2" containerName="revoke-keys"
Oct 30 18:14:29.445882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.445857    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.460131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.460107    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:14:29.460258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.460183    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:14:29.474683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.474658    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:14:29.474844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.474691    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:14:29.477153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.477134    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"]
Oct 30 18:14:29.538226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.538195    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjzs\" (UniqueName: \"kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.538409 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.538378    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.538467 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.538410    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.639870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.639827    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.640052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.639898    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.640052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.639954    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjzs\" (UniqueName: \"kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.640683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.640660    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.642290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.642264    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.662731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.662712    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjzs\" (UniqueName: \"kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs\") pod \"maas-api-revoke-sub-d3fba056-l62z7\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:29.762259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:29.762204    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:30.061339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:30.061141    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"]
Oct 30 18:14:30.488945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:30.488913    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7" event={"ID":"25679fd6-61f0-464b-8537-3a8249c4bf76","Type":"ContainerStarted","Data":"758a1a6d64bdd0d76bf8c701b3ec4280a2bd706d656920a1389b081207581295"}
Oct 30 18:14:30.489376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:30.488969    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7" event={"ID":"25679fd6-61f0-464b-8537-3a8249c4bf76","Type":"ContainerStarted","Data":"7d1ab4c4e4c51f22bc8959e5c9a56aec2766460158aa58381218419fa60e8d25"}
Oct 30 18:14:30.528244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:30.528193    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7" podStartSLOduration=1.528178209 podStartE2EDuration="1.528178209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:30.526998738 +0000 UTC m=+3676.289042552" watchObservedRunningTime="2026-10-30 18:14:30.528178209 +0000 UTC m=+3676.290222056"
Oct 30 18:14:31.507398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:31.506929    5534 generic.go:356] "Generic (PLEG): container finished" podID="25679fd6-61f0-464b-8537-3a8249c4bf76" containerID="758a1a6d64bdd0d76bf8c701b3ec4280a2bd706d656920a1389b081207581295" exitCode=0
Oct 30 18:14:31.507398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:31.507041    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7" event={"ID":"25679fd6-61f0-464b-8537-3a8249c4bf76","Type":"ContainerDied","Data":"758a1a6d64bdd0d76bf8c701b3ec4280a2bd706d656920a1389b081207581295"}
Oct 30 18:14:32.926681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:32.922782    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:14:32.926681 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:32.922849    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:14:33.152670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.151755    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:14:33.152670 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.151822    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:14:33.353135 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.352816    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:33.391451 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.391406    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca\") pod \"25679fd6-61f0-464b-8537-3a8249c4bf76\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") "
Oct 30 18:14:33.413274 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.413218    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs\" (UniqueName: \"kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs\") pod \"25679fd6-61f0-464b-8537-3a8249c4bf76\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") "
Oct 30 18:14:33.414282 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.413511    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca\") pod \"25679fd6-61f0-464b-8537-3a8249c4bf76\" (UID: \"25679fd6-61f0-464b-8537-3a8249c4bf76\") "
Oct 30 18:14:33.414452 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.413913    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca" pod "25679fd6-61f0-464b-8537-3a8249c4bf76" (UID: "25679fd6-61f0-464b-8537-3a8249c4bf76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:14:33.415099 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.415068    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:14:33.433808 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.433764    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs" pod "25679fd6-61f0-464b-8537-3a8249c4bf76" (UID: "25679fd6-61f0-464b-8537-3a8249c4bf76"). InnerVolumeSpecName "kube-api-access-mmjzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:14:33.436190 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.436152    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca" pod "25679fd6-61f0-464b-8537-3a8249c4bf76" (UID: "25679fd6-61f0-464b-8537-3a8249c4bf76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:14:33.522758 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.516579    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25679fd6-61f0-464b-8537-3a8249c4bf76-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:14:33.522758 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.516616    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjzs\" (UniqueName: \"kubernetes.io/projected/25679fd6-61f0-464b-8537-3a8249c4bf76-kube-api-access-mmjzs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:14:33.527242 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.527214    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7" event={"ID":"25679fd6-61f0-464b-8537-3a8249c4bf76","Type":"ContainerDied","Data":"7d1ab4c4e4c51f22bc8959e5c9a56aec2766460158aa58381218419fa60e8d25"}
Oct 30 18:14:33.527430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.527410    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d1ab4c4e4c51f22bc8959e5c9a56aec2766460158aa58381218419fa60e8d25"
Oct 30 18:14:33.527430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.527234    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"
Oct 30 18:14:33.748533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:14:33.748499    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:14:45.244115 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:14:45.244075    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:15:00.148296 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.148214    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"]
Oct 30 18:15:00.148794 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.148696    5534 cpu_manager.go:414] "containerMap: removing container" podUID="25679fd6-61f0-464b-8537-3a8249c4bf76" containerName="revoke-keys"
Oct 30 18:15:00.148794 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.148710    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25679fd6-61f0-464b-8537-3a8249c4bf76" containerName="revoke-keys"
Oct 30 18:15:00.148882 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.148797    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="25679fd6-61f0-464b-8537-3a8249c4bf76" containerName="revoke-keys"
Oct 30 18:15:00.151961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.151944    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.166188 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.166167    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:15:00.178083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.178050    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:15:00.178270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.178148    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:15:00.181124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.181101    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"]
Oct 30 18:15:00.269301 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.269267    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-zh4rp\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.269510 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.269338    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9t6c\" (UniqueName: \"kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c\") pod \"maas-api-key-cleanup-29889735-zh4rp\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.369987 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.369952    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-zh4rp\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.370158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.370009    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9t6c\" (UniqueName: \"kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c\") pod \"maas-api-key-cleanup-29889735-zh4rp\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.370710 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.370675    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-zh4rp\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.392570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.392543    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9t6c\" (UniqueName: \"kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c\") pod \"maas-api-key-cleanup-29889735-zh4rp\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.465282 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.465200    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:00.763837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:00.763812    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"]
Oct 30 18:15:00.766315 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:15:00.766287    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccf672e_befd_4f9e_a93c_ced25de434c5.slice/crio-77c33246c0dfed517b80a782a1e57083f6c033eb66c9ff6a4b25df197893cd50 WatchSource:0}: Error finding container 77c33246c0dfed517b80a782a1e57083f6c033eb66c9ff6a4b25df197893cd50: Status 404 returned error can't find the container with id 77c33246c0dfed517b80a782a1e57083f6c033eb66c9ff6a4b25df197893cd50
Oct 30 18:15:01.670473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:01.670441    5534 generic.go:356] "Generic (PLEG): container finished" podID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerID="6c0b66a23af870f06e0ceb05a5ea92c2791634fdf8d2bca371ee80e5657e26bc" exitCode=60
Oct 30 18:15:01.670888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:01.670524    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" event={"ID":"4ccf672e-befd-4f9e-a93c-ced25de434c5","Type":"ContainerDied","Data":"6c0b66a23af870f06e0ceb05a5ea92c2791634fdf8d2bca371ee80e5657e26bc"}
Oct 30 18:15:01.670888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:01.670573    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" event={"ID":"4ccf672e-befd-4f9e-a93c-ced25de434c5","Type":"ContainerStarted","Data":"77c33246c0dfed517b80a782a1e57083f6c033eb66c9ff6a4b25df197893cd50"}
Oct 30 18:15:01.670888 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:01.670763    5534 scope.go:122] "RemoveContainer" containerID="6c0b66a23af870f06e0ceb05a5ea92c2791634fdf8d2bca371ee80e5657e26bc"
Oct 30 18:15:02.677817 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:02.677781    5534 generic.go:356] "Generic (PLEG): container finished" podID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerID="ede7f3b82c300415cd00469428f224e26a6fd4337258c05c08458b3bfd244272" exitCode=60
Oct 30 18:15:02.678263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:02.677853    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" event={"ID":"4ccf672e-befd-4f9e-a93c-ced25de434c5","Type":"ContainerDied","Data":"ede7f3b82c300415cd00469428f224e26a6fd4337258c05c08458b3bfd244272"}
Oct 30 18:15:02.678263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:02.677899    5534 scope.go:122] "RemoveContainer" containerID="6c0b66a23af870f06e0ceb05a5ea92c2791634fdf8d2bca371ee80e5657e26bc"
Oct 30 18:15:02.678263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:02.678127    5534 scope.go:122] "RemoveContainer" containerID="ede7f3b82c300415cd00469428f224e26a6fd4337258c05c08458b3bfd244272"
Oct 30 18:15:02.678428 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:15:02.678388    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-zh4rp_odh-ai-gateway-infra(4ccf672e-befd-4f9e-a93c-ced25de434c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5"
Oct 30 18:15:03.684946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:03.684912    5534 scope.go:122] "RemoveContainer" containerID="ede7f3b82c300415cd00469428f224e26a6fd4337258c05c08458b3bfd244272"
Oct 30 18:15:03.685445 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:15:03.685179    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-zh4rp_odh-ai-gateway-infra(4ccf672e-befd-4f9e-a93c-ced25de434c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5"
Oct 30 18:15:05.132541 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:05.132497    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:15:14.364941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:14.364908    5534 scope.go:122] "RemoveContainer" containerID="ede7f3b82c300415cd00469428f224e26a6fd4337258c05c08458b3bfd244272"
Oct 30 18:15:14.770828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:14.770787    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" event={"ID":"4ccf672e-befd-4f9e-a93c-ced25de434c5","Type":"ContainerStarted","Data":"32848411ebc231aa23dd36dbbb847005f006abc512f30f9b9b8dd186ce568451"}
Oct 30 18:15:14.809803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:14.809749    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" podStartSLOduration=14.809735213 podStartE2EDuration="14.809735213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:14.808964419 +0000 UTC m=+3720.571008233" watchObservedRunningTime="2026-10-30 18:15:14.809735213 +0000 UTC m=+3720.571779027"
Oct 30 18:15:15.246989 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:15:15.246949    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:15:15.777546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:15.777513    5534 generic.go:356] "Generic (PLEG): container finished" podID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerID="32848411ebc231aa23dd36dbbb847005f006abc512f30f9b9b8dd186ce568451" exitCode=60
Oct 30 18:15:15.777546 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:15.777552    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" event={"ID":"4ccf672e-befd-4f9e-a93c-ced25de434c5","Type":"ContainerDied","Data":"32848411ebc231aa23dd36dbbb847005f006abc512f30f9b9b8dd186ce568451"}
Oct 30 18:15:15.778169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:15.777601    5534 scope.go:122] "RemoveContainer" containerID="ede7f3b82c300415cd00469428f224e26a6fd4337258c05c08458b3bfd244272"
Oct 30 18:15:15.778169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:15.778039    5534 scope.go:122] "RemoveContainer" containerID="32848411ebc231aa23dd36dbbb847005f006abc512f30f9b9b8dd186ce568451"
Oct 30 18:15:15.778374 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:15:15.778350    5534 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-zh4rp_odh-ai-gateway-infra(4ccf672e-befd-4f9e-a93c-ced25de434c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5"
Oct 30 18:15:15.824440 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:15.824401    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"]
Oct 30 18:15:16.926477 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:16.926448    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:17.089708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.089573    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c\" (UniqueName: \"kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c\") pod \"4ccf672e-befd-4f9e-a93c-ced25de434c5\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") "
Oct 30 18:15:17.089708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.089694    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca\") pod \"4ccf672e-befd-4f9e-a93c-ced25de434c5\" (UID: \"4ccf672e-befd-4f9e-a93c-ced25de434c5\") "
Oct 30 18:15:17.090065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.090038    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca" pod "4ccf672e-befd-4f9e-a93c-ced25de434c5" (UID: "4ccf672e-befd-4f9e-a93c-ced25de434c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:15:17.092178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.092156    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c" pod "4ccf672e-befd-4f9e-a93c-ced25de434c5" (UID: "4ccf672e-befd-4f9e-a93c-ced25de434c5"). InnerVolumeSpecName "kube-api-access-f9t6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:15:17.190803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.190760    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9t6c\" (UniqueName: \"kubernetes.io/projected/4ccf672e-befd-4f9e-a93c-ced25de434c5-kube-api-access-f9t6c\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:15:17.190803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.190796    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf672e-befd-4f9e-a93c-ced25de434c5-maas-api-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:15:17.790902 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.790871    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"
Oct 30 18:15:17.791145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.790904    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp" event={"ID":"4ccf672e-befd-4f9e-a93c-ced25de434c5","Type":"ContainerDied","Data":"77c33246c0dfed517b80a782a1e57083f6c033eb66c9ff6a4b25df197893cd50"}
Oct 30 18:15:17.791145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.790965    5534 scope.go:122] "RemoveContainer" containerID="32848411ebc231aa23dd36dbbb847005f006abc512f30f9b9b8dd186ce568451"
Oct 30 18:15:17.838885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.838858    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"]
Oct 30 18:15:17.850378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:17.850356    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-zh4rp"]
Oct 30 18:15:18.151026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:18.150996    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:15:18.368521 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:18.368478    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" path="/var/lib/kubelet/pods/4ccf672e-befd-4f9e-a93c-ced25de434c5/volumes"
Oct 30 18:15:19.007770 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:19.007730    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:15:21.406011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:21.405979    5534 scope.go:122] "RemoveContainer" containerID="bea9db576ff743103d81b72dc63710b32e9bb1439f961d9f3a2f8a05a0eefecf"
Oct 30 18:15:21.415285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:21.415260    5534 scope.go:122] "RemoveContainer" containerID="43350c2424b173009b7414c7a247e2166020e0cc91e4fbb26d79ac4f0316efc9"
Oct 30 18:15:21.424007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:21.423984    5534 scope.go:122] "RemoveContainer" containerID="ac6083522c1c84ed4096bc0b0b8c7f6bccc00c0732c3e13d7475ac14bdcace41"
Oct 30 18:15:28.004828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:28.004774    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:15:28.005487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:28.005450    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:15:32.692787 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.692753    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"]
Oct 30 18:15:32.693185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693161    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693174    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693207    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693213    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693220    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693225    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693305    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.693394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.693315    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:15:32.698157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.698132    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.707555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.707530    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.707710 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.707575    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z7p\" (UniqueName: \"kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.707757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.707713    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.714273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.714240    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:15:32.726025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.725988    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:15:32.728085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.728059    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"]
Oct 30 18:15:32.740707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.740676    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:15:32.740892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.740712    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:15:32.808784 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.808743    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.808974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.808808    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z7p\" (UniqueName: \"kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.808974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.808891    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.809512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.809482    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.811441 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.811421    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:32.831435 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:32.831403    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z7p\" (UniqueName: \"kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p\") pod \"maas-api-revoke-sub-a4aef7d7-nhxrp\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:33.011622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:33.011537    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:33.312178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:33.312088    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"]
Oct 30 18:15:33.320524 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:15:33.320486    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50120b3f_5e49_4d0d_a34c_81835091dd34.slice/crio-735e854ba784a2ba19dfa173918ee5ebe78cbab71e553b21dfa8d6519acf0850 WatchSource:0}: Error finding container 735e854ba784a2ba19dfa173918ee5ebe78cbab71e553b21dfa8d6519acf0850: Status 404 returned error can't find the container with id 735e854ba784a2ba19dfa173918ee5ebe78cbab71e553b21dfa8d6519acf0850
Oct 30 18:15:33.863694 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:33.863658    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:15:33.890739 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:33.890700    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp" event={"ID":"50120b3f-5e49-4d0d-a34c-81835091dd34","Type":"ContainerStarted","Data":"a36ae3c00e0fde055b6417b39d5730353785a1b33c891b5748dc48390e5ce305"}
Oct 30 18:15:33.890893 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:33.890767    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp" event={"ID":"50120b3f-5e49-4d0d-a34c-81835091dd34","Type":"ContainerStarted","Data":"735e854ba784a2ba19dfa173918ee5ebe78cbab71e553b21dfa8d6519acf0850"}
Oct 30 18:15:33.930587 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:33.930526    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp" podStartSLOduration=1.9305070020000001 podStartE2EDuration="1.930507002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:33.929616645 +0000 UTC m=+3739.691660461" watchObservedRunningTime="2026-10-30 18:15:33.930507002 +0000 UTC m=+3739.692550816"
Oct 30 18:15:34.898088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:34.898052    5534 generic.go:356] "Generic (PLEG): container finished" podID="50120b3f-5e49-4d0d-a34c-81835091dd34" containerID="a36ae3c00e0fde055b6417b39d5730353785a1b33c891b5748dc48390e5ce305" exitCode=0
Oct 30 18:15:34.898569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:34.898134    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp" event={"ID":"50120b3f-5e49-4d0d-a34c-81835091dd34","Type":"ContainerDied","Data":"a36ae3c00e0fde055b6417b39d5730353785a1b33c891b5748dc48390e5ce305"}
Oct 30 18:15:36.045906 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.045881    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:36.143824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.143794    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca\") pod \"50120b3f-5e49-4d0d-a34c-81835091dd34\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") "
Oct 30 18:15:36.143824 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.143835    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca\") pod \"50120b3f-5e49-4d0d-a34c-81835091dd34\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") "
Oct 30 18:15:36.144123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.143873    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p\" (UniqueName: \"kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p\") pod \"50120b3f-5e49-4d0d-a34c-81835091dd34\" (UID: \"50120b3f-5e49-4d0d-a34c-81835091dd34\") "
Oct 30 18:15:36.144250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.144229    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca" pod "50120b3f-5e49-4d0d-a34c-81835091dd34" (UID: "50120b3f-5e49-4d0d-a34c-81835091dd34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:15:36.146237 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.146206    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca" pod "50120b3f-5e49-4d0d-a34c-81835091dd34" (UID: "50120b3f-5e49-4d0d-a34c-81835091dd34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:15:36.146351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.146268    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p" pod "50120b3f-5e49-4d0d-a34c-81835091dd34" (UID: "50120b3f-5e49-4d0d-a34c-81835091dd34"). InnerVolumeSpecName "kube-api-access-g5z7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:15:36.244976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.244890    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:15:36.244976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.244920    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50120b3f-5e49-4d0d-a34c-81835091dd34-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:15:36.244976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.244930    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5z7p\" (UniqueName: \"kubernetes.io/projected/50120b3f-5e49-4d0d-a34c-81835091dd34-kube-api-access-g5z7p\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:15:36.910338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.910301    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp" event={"ID":"50120b3f-5e49-4d0d-a34c-81835091dd34","Type":"ContainerDied","Data":"735e854ba784a2ba19dfa173918ee5ebe78cbab71e553b21dfa8d6519acf0850"}
Oct 30 18:15:36.910506 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.910359    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735e854ba784a2ba19dfa173918ee5ebe78cbab71e553b21dfa8d6519acf0850"
Oct 30 18:15:36.910506 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:36.910309    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"
Oct 30 18:15:45.244282 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:15:45.244245    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:15:55.266178 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:15:55.266137    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:16:01.620328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.620292    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"]
Oct 30 18:16:01.620744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.620713    5534 cpu_manager.go:414] "containerMap: removing container" podUID="50120b3f-5e49-4d0d-a34c-81835091dd34" containerName="revoke-keys"
Oct 30 18:16:01.620744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.620724    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50120b3f-5e49-4d0d-a34c-81835091dd34" containerName="revoke-keys"
Oct 30 18:16:01.620825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.620786    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccf672e-befd-4f9e-a93c-ced25de434c5" containerName="cleanup"
Oct 30 18:16:01.620825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.620801    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="50120b3f-5e49-4d0d-a34c-81835091dd34" containerName="revoke-keys"
Oct 30 18:16:01.623889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.623870    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.638140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.638110    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:16:01.652946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.652911    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:16:01.656019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.655976    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"]
Oct 30 18:16:01.714997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.714961    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:16:01.714997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.714977    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:16:01.753813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.753775    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.753994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.753837    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.754064 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.754030    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qph4\" (UniqueName: \"kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.854983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.854950    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qph4\" (UniqueName: \"kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.855199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.855027    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.855199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.855065    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.855746 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.855719    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.857582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.857547    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.877745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.877679    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qph4\" (UniqueName: \"kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4\") pod \"maas-api-revoke-sub-79805ca9-jn6lq\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:01.936294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:01.936255    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:02.235430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:02.235403    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"]
Oct 30 18:16:02.237886 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:16:02.237858    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575fffa3_f8cc_4f28_8770_ac9912224d40.slice/crio-a210e26ecfb1cdced064d070bf437bf8cddec4c5218eaf19fdf73c42ff9243ae WatchSource:0}: Error finding container a210e26ecfb1cdced064d070bf437bf8cddec4c5218eaf19fdf73c42ff9243ae: Status 404 returned error can't find the container with id a210e26ecfb1cdced064d070bf437bf8cddec4c5218eaf19fdf73c42ff9243ae
Oct 30 18:16:03.098959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:03.098895    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq" event={"ID":"575fffa3-f8cc-4f28-8770-ac9912224d40","Type":"ContainerStarted","Data":"66f9d96016fa5631c5f6a30d884f54741338c4106d7bb4a2aba1547cbcffaca9"}
Oct 30 18:16:03.099394 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:03.099048    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq" event={"ID":"575fffa3-f8cc-4f28-8770-ac9912224d40","Type":"ContainerStarted","Data":"a210e26ecfb1cdced064d070bf437bf8cddec4c5218eaf19fdf73c42ff9243ae"}
Oct 30 18:16:03.138544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:03.138476    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq" podStartSLOduration=2.138456263 podStartE2EDuration="2.138456263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:03.13756161 +0000 UTC m=+3768.899605434" watchObservedRunningTime="2026-10-30 18:16:03.138456263 +0000 UTC m=+3768.900500078"
Oct 30 18:16:04.106564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:04.106529    5534 generic.go:356] "Generic (PLEG): container finished" podID="575fffa3-f8cc-4f28-8770-ac9912224d40" containerID="66f9d96016fa5631c5f6a30d884f54741338c4106d7bb4a2aba1547cbcffaca9" exitCode=0
Oct 30 18:16:04.107007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:04.106616    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq" event={"ID":"575fffa3-f8cc-4f28-8770-ac9912224d40","Type":"ContainerDied","Data":"66f9d96016fa5631c5f6a30d884f54741338c4106d7bb4a2aba1547cbcffaca9"}
Oct 30 18:16:05.249632 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.249603    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:05.254254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.254221    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:16:05.393717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.393620    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca\") pod \"575fffa3-f8cc-4f28-8770-ac9912224d40\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") "
Oct 30 18:16:05.393717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.393686    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca\") pod \"575fffa3-f8cc-4f28-8770-ac9912224d40\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") "
Oct 30 18:16:05.393977 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.393793    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4\" (UniqueName: \"kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4\") pod \"575fffa3-f8cc-4f28-8770-ac9912224d40\" (UID: \"575fffa3-f8cc-4f28-8770-ac9912224d40\") "
Oct 30 18:16:05.394079 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.394053    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca" pod "575fffa3-f8cc-4f28-8770-ac9912224d40" (UID: "575fffa3-f8cc-4f28-8770-ac9912224d40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:16:05.396046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.396024    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4" pod "575fffa3-f8cc-4f28-8770-ac9912224d40" (UID: "575fffa3-f8cc-4f28-8770-ac9912224d40"). InnerVolumeSpecName "kube-api-access-6qph4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:16:05.396139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.396042    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca" pod "575fffa3-f8cc-4f28-8770-ac9912224d40" (UID: "575fffa3-f8cc-4f28-8770-ac9912224d40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:16:05.495038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.494996    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qph4\" (UniqueName: \"kubernetes.io/projected/575fffa3-f8cc-4f28-8770-ac9912224d40-kube-api-access-6qph4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:05.495038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.495028    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:05.495038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:05.495038    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575fffa3-f8cc-4f28-8770-ac9912224d40-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:06.118481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:06.118453    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"
Oct 30 18:16:06.118481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:06.118468    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq" event={"ID":"575fffa3-f8cc-4f28-8770-ac9912224d40","Type":"ContainerDied","Data":"a210e26ecfb1cdced064d070bf437bf8cddec4c5218eaf19fdf73c42ff9243ae"}
Oct 30 18:16:06.118835 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:06.118518    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a210e26ecfb1cdced064d070bf437bf8cddec4c5218eaf19fdf73c42ff9243ae"
Oct 30 18:16:12.028839 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:12.028801    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"]
Oct 30 18:16:12.040921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:12.040894    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"]
Oct 30 18:16:12.042561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:12.042540    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c7158d-8wdjn"]
Oct 30 18:16:12.044027 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:12.043992    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b646be53-b9rp5"]
Oct 30 18:16:12.367818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:12.367785    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01deb567-0ca4-47f9-8118-b26b478a4226" path="/var/lib/kubelet/pods/01deb567-0ca4-47f9-8118-b26b478a4226/volumes"
Oct 30 18:16:12.368306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:12.368277    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb76301d-6e71-4ae3-9dbd-dc58de3d81e6" path="/var/lib/kubelet/pods/fb76301d-6e71-4ae3-9dbd-dc58de3d81e6/volumes"
Oct 30 18:16:15.431934 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:16:15.431835    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:16:19.524236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.524198    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"]
Oct 30 18:16:19.524859 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.524837    5534 cpu_manager.go:414] "containerMap: removing container" podUID="575fffa3-f8cc-4f28-8770-ac9912224d40" containerName="revoke-keys"
Oct 30 18:16:19.524950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.524863    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575fffa3-f8cc-4f28-8770-ac9912224d40" containerName="revoke-keys"
Oct 30 18:16:19.525009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.524976    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="575fffa3-f8cc-4f28-8770-ac9912224d40" containerName="revoke-keys"
Oct 30 18:16:19.528107 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.528086    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.542232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.542196    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:16:19.554086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.554048    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:16:19.554086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.554076    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:16:19.556309 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.556282    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"]
Oct 30 18:16:19.608233 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.608197    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.608430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.608310    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.608430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.608362    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6d4\" (UniqueName: \"kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.643457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.643426    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:16:19.716671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.709602    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.716671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.709771    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.716671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.709841    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6d4\" (UniqueName: \"kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.716671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.711129    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.716671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.714930    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.732675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.732624    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6d4\" (UniqueName: \"kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4\") pod \"maas-api-revoke-sub-0b7e8edb-clk8x\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:19.841378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:19.841350    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:20.140563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:20.140536    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"]
Oct 30 18:16:20.142941 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:16:20.142914    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f847b91_8daa_4dc4_b93e_023abb5969a6.slice/crio-015e3910d3f0f5d17a1beb581f2fd7bd686a58e847715929fc5aa403c8809724 WatchSource:0}: Error finding container 015e3910d3f0f5d17a1beb581f2fd7bd686a58e847715929fc5aa403c8809724: Status 404 returned error can't find the container with id 015e3910d3f0f5d17a1beb581f2fd7bd686a58e847715929fc5aa403c8809724
Oct 30 18:16:20.260624 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:20.260588    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x" event={"ID":"5f847b91-8daa-4dc4-b93e-023abb5969a6","Type":"ContainerStarted","Data":"015e3910d3f0f5d17a1beb581f2fd7bd686a58e847715929fc5aa403c8809724"}
Oct 30 18:16:21.268327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:21.268233    5534 generic.go:356] "Generic (PLEG): container finished" podID="5f847b91-8daa-4dc4-b93e-023abb5969a6" containerID="673c2637ffad6e69af2c94ccec2d6916a2f7985e5908dc2e02766ac6659bbe19" exitCode=0
Oct 30 18:16:21.268739 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:21.268321    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x" event={"ID":"5f847b91-8daa-4dc4-b93e-023abb5969a6","Type":"ContainerDied","Data":"673c2637ffad6e69af2c94ccec2d6916a2f7985e5908dc2e02766ac6659bbe19"}
Oct 30 18:16:21.458081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:21.458049    5534 scope.go:122] "RemoveContainer" containerID="2806600eb9ab00098fc061171914b38bcf01612442ce4777d52e74ba6a700896"
Oct 30 18:16:22.402695 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.402670    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:22.432457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.432426    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4\" (UniqueName: \"kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4\") pod \"5f847b91-8daa-4dc4-b93e-023abb5969a6\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") "
Oct 30 18:16:22.432612 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.432501    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca\") pod \"5f847b91-8daa-4dc4-b93e-023abb5969a6\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") "
Oct 30 18:16:22.432612 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.432554    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca\") pod \"5f847b91-8daa-4dc4-b93e-023abb5969a6\" (UID: \"5f847b91-8daa-4dc4-b93e-023abb5969a6\") "
Oct 30 18:16:22.432881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.432857    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca" pod "5f847b91-8daa-4dc4-b93e-023abb5969a6" (UID: "5f847b91-8daa-4dc4-b93e-023abb5969a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:16:22.434792 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.434766    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca" pod "5f847b91-8daa-4dc4-b93e-023abb5969a6" (UID: "5f847b91-8daa-4dc4-b93e-023abb5969a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:16:22.434960 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.434941    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4" pod "5f847b91-8daa-4dc4-b93e-023abb5969a6" (UID: "5f847b91-8daa-4dc4-b93e-023abb5969a6"). InnerVolumeSpecName "kube-api-access-8m6d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:16:22.533363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.533272    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6d4\" (UniqueName: \"kubernetes.io/projected/5f847b91-8daa-4dc4-b93e-023abb5969a6-kube-api-access-8m6d4\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:22.533363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.533303    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:22.533363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:22.533313    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f847b91-8daa-4dc4-b93e-023abb5969a6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:23.279241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:23.279205    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x" event={"ID":"5f847b91-8daa-4dc4-b93e-023abb5969a6","Type":"ContainerDied","Data":"015e3910d3f0f5d17a1beb581f2fd7bd686a58e847715929fc5aa403c8809724"}
Oct 30 18:16:23.279428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:23.279257    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015e3910d3f0f5d17a1beb581f2fd7bd686a58e847715929fc5aa403c8809724"
Oct 30 18:16:23.279428 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:23.279218    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"
Oct 30 18:16:30.270667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.270554    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"]
Oct 30 18:16:30.271077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.271005    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5f847b91-8daa-4dc4-b93e-023abb5969a6" containerName="revoke-keys"
Oct 30 18:16:30.271077 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.271018    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f847b91-8daa-4dc4-b93e-023abb5969a6" containerName="revoke-keys"
Oct 30 18:16:30.271177 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.271119    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f847b91-8daa-4dc4-b93e-023abb5969a6" containerName="revoke-keys"
Oct 30 18:16:30.274461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.274437    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.288735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.288706    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:16:30.300833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.300785    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:16:30.302553 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.302526    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"]
Oct 30 18:16:30.309038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.309012    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.309195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.309068    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.309195 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.309131    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8m2\" (UniqueName: \"kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.361715 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.361683    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:16:30.361887 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.361731    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:16:30.410234 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.410196    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.410450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.410279    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.410450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.410307    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8m2\" (UniqueName: \"kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.411300 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.411264    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.412755 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.412734    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.433044 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.433012    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8m2\" (UniqueName: \"kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2\") pod \"maas-api-revoke-sub-3d4156f1-96466\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.589109 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.589027    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:30.892014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:30.891978    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"]
Oct 30 18:16:30.896798 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:16:30.896765    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89295a3_2eb5_41c9_9c58_f638cef7650a.slice/crio-30df92fc6b5eea701809ba3b48b7475452ce42fe7eba34a5f8da14729b094c98 WatchSource:0}: Error finding container 30df92fc6b5eea701809ba3b48b7475452ce42fe7eba34a5f8da14729b094c98: Status 404 returned error can't find the container with id 30df92fc6b5eea701809ba3b48b7475452ce42fe7eba34a5f8da14729b094c98
Oct 30 18:16:31.334494 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:31.334445    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466" event={"ID":"d89295a3-2eb5-41c9-9c58-f638cef7650a","Type":"ContainerStarted","Data":"f002d0087bfc4a0227da792e46f81cb8de52ef310ef507caba8effb645353ee2"}
Oct 30 18:16:31.334950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:31.334513    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466" event={"ID":"d89295a3-2eb5-41c9-9c58-f638cef7650a","Type":"ContainerStarted","Data":"30df92fc6b5eea701809ba3b48b7475452ce42fe7eba34a5f8da14729b094c98"}
Oct 30 18:16:31.377078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:31.377028    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466" podStartSLOduration=1.377012058 podStartE2EDuration="1.377012058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:31.375346598 +0000 UTC m=+3797.137390412" watchObservedRunningTime="2026-10-30 18:16:31.377012058 +0000 UTC m=+3797.139055870"
Oct 30 18:16:32.345305 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:32.345268    5534 generic.go:356] "Generic (PLEG): container finished" podID="d89295a3-2eb5-41c9-9c58-f638cef7650a" containerID="f002d0087bfc4a0227da792e46f81cb8de52ef310ef507caba8effb645353ee2" exitCode=0
Oct 30 18:16:32.345734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:32.345362    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466" event={"ID":"d89295a3-2eb5-41c9-9c58-f638cef7650a","Type":"ContainerDied","Data":"f002d0087bfc4a0227da792e46f81cb8de52ef310ef507caba8effb645353ee2"}
Oct 30 18:16:33.508727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.508702    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:33.540903 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.540869    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2\" (UniqueName: \"kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2\") pod \"d89295a3-2eb5-41c9-9c58-f638cef7650a\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") "
Oct 30 18:16:33.541094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.541016    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca\") pod \"d89295a3-2eb5-41c9-9c58-f638cef7650a\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") "
Oct 30 18:16:33.541094 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.541054    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca\") pod \"d89295a3-2eb5-41c9-9c58-f638cef7650a\" (UID: \"d89295a3-2eb5-41c9-9c58-f638cef7650a\") "
Oct 30 18:16:33.541489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.541451    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca" pod "d89295a3-2eb5-41c9-9c58-f638cef7650a" (UID: "d89295a3-2eb5-41c9-9c58-f638cef7650a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:16:33.543114 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.543079    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2" pod "d89295a3-2eb5-41c9-9c58-f638cef7650a" (UID: "d89295a3-2eb5-41c9-9c58-f638cef7650a"). InnerVolumeSpecName "kube-api-access-vh8m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:16:33.543211 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.543154    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca" pod "d89295a3-2eb5-41c9-9c58-f638cef7650a" (UID: "d89295a3-2eb5-41c9-9c58-f638cef7650a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:16:33.642398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.642309    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:33.642398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.642348    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89295a3-2eb5-41c9-9c58-f638cef7650a-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:33.642398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.642359    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8m2\" (UniqueName: \"kubernetes.io/projected/d89295a3-2eb5-41c9-9c58-f638cef7650a-kube-api-access-vh8m2\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:33.987991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:33.987904    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:16:34.358314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:34.358275    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466" event={"ID":"d89295a3-2eb5-41c9-9c58-f638cef7650a","Type":"ContainerDied","Data":"30df92fc6b5eea701809ba3b48b7475452ce42fe7eba34a5f8da14729b094c98"}
Oct 30 18:16:34.358540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:34.358354    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30df92fc6b5eea701809ba3b48b7475452ce42fe7eba34a5f8da14729b094c98"
Oct 30 18:16:34.358540 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:34.358292    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"
Oct 30 18:16:45.244518 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:16:45.244474    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:16:51.639777 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.639739    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"]
Oct 30 18:16:51.651564 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.651537    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96466"]
Oct 30 18:16:51.690216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.690189    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"]
Oct 30 18:16:51.690601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.690583    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d89295a3-2eb5-41c9-9c58-f638cef7650a" containerName="revoke-keys"
Oct 30 18:16:51.690601 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.690599    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89295a3-2eb5-41c9-9c58-f638cef7650a" containerName="revoke-keys"
Oct 30 18:16:51.690788 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.690696    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89295a3-2eb5-41c9-9c58-f638cef7650a" containerName="revoke-keys"
Oct 30 18:16:51.693727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.693708    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.708017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.707990    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:16:51.708017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.707993    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:16:51.708201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.708096    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:16:51.708201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.708130    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:16:51.714497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.714460    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"]
Oct 30 18:16:51.795261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.795227    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.795462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.795285    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xrx\" (UniqueName: \"kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.795462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.795344    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.896248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.896162    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.896248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.896220    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xrx\" (UniqueName: \"kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.896465 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.896319    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.896879 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.896857    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.898873 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.898849    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:51.919442 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:51.919408    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xrx\" (UniqueName: \"kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx\") pod \"maas-api-revoke-sub-3d4156f1-vmqw5\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:52.006528 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:52.006489    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:52.305286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:52.305259    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"]
Oct 30 18:16:52.307155 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:16:52.307108    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af3c819_ed7f_45d0_9f53_6ea52723d9c6.slice/crio-27562c303db97a55988f27907ab612ec5a2ffdb13d14a381ac79aeea48c852f6 WatchSource:0}: Error finding container 27562c303db97a55988f27907ab612ec5a2ffdb13d14a381ac79aeea48c852f6: Status 404 returned error can't find the container with id 27562c303db97a55988f27907ab612ec5a2ffdb13d14a381ac79aeea48c852f6
Oct 30 18:16:52.368876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:52.368847    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89295a3-2eb5-41c9-9c58-f638cef7650a" path="/var/lib/kubelet/pods/d89295a3-2eb5-41c9-9c58-f638cef7650a/volumes"
Oct 30 18:16:52.462064 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:52.462028    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5" event={"ID":"0af3c819-ed7f-45d0-9f53-6ea52723d9c6","Type":"ContainerStarted","Data":"27562c303db97a55988f27907ab612ec5a2ffdb13d14a381ac79aeea48c852f6"}
Oct 30 18:16:53.468197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:53.468162    5534 generic.go:356] "Generic (PLEG): container finished" podID="0af3c819-ed7f-45d0-9f53-6ea52723d9c6" containerID="1cf2521d2b67780da6626f7d69e559ed4847c6c4202bc810a8f0b7f2dcd266c9" exitCode=0
Oct 30 18:16:53.468744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:53.468237    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5" event={"ID":"0af3c819-ed7f-45d0-9f53-6ea52723d9c6","Type":"ContainerDied","Data":"1cf2521d2b67780da6626f7d69e559ed4847c6c4202bc810a8f0b7f2dcd266c9"}
Oct 30 18:16:54.611411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.611382    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:54.721334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.721300    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca\") pod \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") "
Oct 30 18:16:54.721557 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.721377    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca\") pod \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") "
Oct 30 18:16:54.721557 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.721409    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx\" (UniqueName: \"kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx\") pod \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\" (UID: \"0af3c819-ed7f-45d0-9f53-6ea52723d9c6\") "
Oct 30 18:16:54.721801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.721776    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca" pod "0af3c819-ed7f-45d0-9f53-6ea52723d9c6" (UID: "0af3c819-ed7f-45d0-9f53-6ea52723d9c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:16:54.723759 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.723702    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx" pod "0af3c819-ed7f-45d0-9f53-6ea52723d9c6" (UID: "0af3c819-ed7f-45d0-9f53-6ea52723d9c6"). InnerVolumeSpecName "kube-api-access-59xrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:16:54.723759 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.723717    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca" pod "0af3c819-ed7f-45d0-9f53-6ea52723d9c6" (UID: "0af3c819-ed7f-45d0-9f53-6ea52723d9c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:16:54.823009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.822971    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:54.823009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.823003    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xrx\" (UniqueName: \"kubernetes.io/projected/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-kube-api-access-59xrx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:54.823009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:54.823014    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af3c819-ed7f-45d0-9f53-6ea52723d9c6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:16:55.444772 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:55.444741    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:16:55.479172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:55.479145    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"
Oct 30 18:16:55.479327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:55.479180    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5" event={"ID":"0af3c819-ed7f-45d0-9f53-6ea52723d9c6","Type":"ContainerDied","Data":"27562c303db97a55988f27907ab612ec5a2ffdb13d14a381ac79aeea48c852f6"}
Oct 30 18:16:55.479327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:16:55.479230    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27562c303db97a55988f27907ab612ec5a2ffdb13d14a381ac79aeea48c852f6"
Oct 30 18:17:02.027961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:02.027919    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"]
Oct 30 18:17:02.038595 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:02.038560    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xczmv"]
Oct 30 18:17:02.368068 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:02.368033    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1528f365-305b-45ca-bc80-30aaea66cd87" path="/var/lib/kubelet/pods/1528f365-305b-45ca-bc80-30aaea66cd87/volumes"
Oct 30 18:17:15.244542 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:17:15.244503    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:17:15.416149 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.416105    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"]
Oct 30 18:17:15.427376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.427351    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vmqw5"]
Oct 30 18:17:15.460327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.460297    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"]
Oct 30 18:17:15.460727 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.460712    5534 cpu_manager.go:414] "containerMap: removing container" podUID="0af3c819-ed7f-45d0-9f53-6ea52723d9c6" containerName="revoke-keys"
Oct 30 18:17:15.460786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.460728    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af3c819-ed7f-45d0-9f53-6ea52723d9c6" containerName="revoke-keys"
Oct 30 18:17:15.460829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.460806    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af3c819-ed7f-45d0-9f53-6ea52723d9c6" containerName="revoke-keys"
Oct 30 18:17:15.464996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.464977    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.482811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.482275    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:17:15.482811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.482305    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:17:15.482811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.482306    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:17:15.482811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.482381    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:17:15.483885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.483860    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"]
Oct 30 18:17:15.509426 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.509338    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.509586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.509437    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgjz\" (UniqueName: \"kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.509698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.509577    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.610461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.610431    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.610714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.610495    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.610714 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.610518    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgjz\" (UniqueName: \"kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.611286 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.611257    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.612925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.612904    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.633214 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.633194    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgjz\" (UniqueName: \"kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz\") pod \"maas-api-revoke-sub-3d4156f1-h6t65\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:15.777816 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:15.777734    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:16.076366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:16.076342    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"]
Oct 30 18:17:16.078429 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:17:16.078388    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa43c06_06f0_4a10_bd23_8790990fb139.slice/crio-21924d7467625214e18c04f51dd9c875f66e143fefd6720ba1b331bed62459fc WatchSource:0}: Error finding container 21924d7467625214e18c04f51dd9c875f66e143fefd6720ba1b331bed62459fc: Status 404 returned error can't find the container with id 21924d7467625214e18c04f51dd9c875f66e143fefd6720ba1b331bed62459fc
Oct 30 18:17:16.368164 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:16.368135    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af3c819-ed7f-45d0-9f53-6ea52723d9c6" path="/var/lib/kubelet/pods/0af3c819-ed7f-45d0-9f53-6ea52723d9c6/volumes"
Oct 30 18:17:16.593786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:16.593744    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65" event={"ID":"2aa43c06-06f0-4a10-bd23-8790990fb139","Type":"ContainerStarted","Data":"215e3cb1cc714fea04e51498fd25f9335ea0e1c7d68f115b91c707df6f1d0612"}
Oct 30 18:17:16.593993 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:16.593805    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65" event={"ID":"2aa43c06-06f0-4a10-bd23-8790990fb139","Type":"ContainerStarted","Data":"21924d7467625214e18c04f51dd9c875f66e143fefd6720ba1b331bed62459fc"}
Oct 30 18:17:16.634250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:16.634166    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65" podStartSLOduration=1.634151645 podStartE2EDuration="1.634151645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:16.632973212 +0000 UTC m=+3842.395017028" watchObservedRunningTime="2026-10-30 18:17:16.634151645 +0000 UTC m=+3842.396195485"
Oct 30 18:17:17.601951 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:17.601918    5534 generic.go:356] "Generic (PLEG): container finished" podID="2aa43c06-06f0-4a10-bd23-8790990fb139" containerID="215e3cb1cc714fea04e51498fd25f9335ea0e1c7d68f115b91c707df6f1d0612" exitCode=0
Oct 30 18:17:17.602397 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:17.602010    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65" event={"ID":"2aa43c06-06f0-4a10-bd23-8790990fb139","Type":"ContainerDied","Data":"215e3cb1cc714fea04e51498fd25f9335ea0e1c7d68f115b91c707df6f1d0612"}
Oct 30 18:17:18.743458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.743432    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:18.843283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.843255    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca\") pod \"2aa43c06-06f0-4a10-bd23-8790990fb139\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") "
Oct 30 18:17:18.843283 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.843296    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz\" (UniqueName: \"kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz\") pod \"2aa43c06-06f0-4a10-bd23-8790990fb139\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") "
Oct 30 18:17:18.843563 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.843329    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca\") pod \"2aa43c06-06f0-4a10-bd23-8790990fb139\" (UID: \"2aa43c06-06f0-4a10-bd23-8790990fb139\") "
Oct 30 18:17:18.843678 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.843625    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca" pod "2aa43c06-06f0-4a10-bd23-8790990fb139" (UID: "2aa43c06-06f0-4a10-bd23-8790990fb139"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:17:18.845575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.845543    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca" pod "2aa43c06-06f0-4a10-bd23-8790990fb139" (UID: "2aa43c06-06f0-4a10-bd23-8790990fb139"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:17:18.845750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.845728    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz" pod "2aa43c06-06f0-4a10-bd23-8790990fb139" (UID: "2aa43c06-06f0-4a10-bd23-8790990fb139"). InnerVolumeSpecName "kube-api-access-gmgjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:17:18.945366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.945276    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:17:18.945366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.945307    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgjz\" (UniqueName: \"kubernetes.io/projected/2aa43c06-06f0-4a10-bd23-8790990fb139-kube-api-access-gmgjz\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:17:18.945366 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:18.945318    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa43c06-06f0-4a10-bd23-8790990fb139-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:17:19.612881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:19.612847    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"
Oct 30 18:17:19.613067 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:19.612877    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65" event={"ID":"2aa43c06-06f0-4a10-bd23-8790990fb139","Type":"ContainerDied","Data":"21924d7467625214e18c04f51dd9c875f66e143fefd6720ba1b331bed62459fc"}
Oct 30 18:17:19.613067 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:19.612928    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21924d7467625214e18c04f51dd9c875f66e143fefd6720ba1b331bed62459fc"
Oct 30 18:17:21.479092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:21.479059    5534 scope.go:122] "RemoveContainer" containerID="1898fe33ef7626587c327e5eeb49a5a337651eb888c3a92196bb697508e4ed63"
Oct 30 18:17:21.488550 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:21.488533    5534 scope.go:122] "RemoveContainer" containerID="8a9ebd76515303d2d75aa1009a4b12e5d5a1c09872cedda4ccfa777d32bcc3e7"
Oct 30 18:17:21.496169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:21.496147    5534 scope.go:122] "RemoveContainer" containerID="0910890d5f755f08e5439dfe26e1e3de2a68f941eaf6a12d721854ac6684fc19"
Oct 30 18:17:34.135623 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:34.135569    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:40.780126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:40.780076    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:41.301222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:41.301192    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:41.577561 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:41.577466    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:44.643990 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:44.643951    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:44.844400 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:44.844364    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:45.247034 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:17:45.246989    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:17:46.071914 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.071876    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"]
Oct 30 18:17:46.072433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.072328    5534 cpu_manager.go:414] "containerMap: removing container" podUID="2aa43c06-06f0-4a10-bd23-8790990fb139" containerName="revoke-keys"
Oct 30 18:17:46.072433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.072342    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa43c06-06f0-4a10-bd23-8790990fb139" containerName="revoke-keys"
Oct 30 18:17:46.072433 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.072419    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa43c06-06f0-4a10-bd23-8790990fb139" containerName="revoke-keys"
Oct 30 18:17:46.077620 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.077592    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.091955 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.091931    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:17:46.092091 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.091965    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:17:46.104182 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.104154    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s8xph\""
Oct 30 18:17:46.104750 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.104715    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-05e969-serving-cert\""
Oct 30 18:17:46.106703 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.106681    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"]
Oct 30 18:17:46.162019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.161989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:46.167805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.167784    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:17:46.213571 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.213538    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.213806 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.213707    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.213885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.213807    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.213969 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.213946    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.315548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.315514    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.315742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.315569    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.315742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.315623    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.315742 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.315698    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.316223 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.316195    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.318251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.318226    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.318365 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.318281    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.337932 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.337868    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76\") pod \"maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.391031 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.390993    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:46.391737 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.391709    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:46.451343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.451312    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"]
Oct 30 18:17:46.456929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.456901    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.474624 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.474559    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"]
Oct 30 18:17:46.507939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.507902    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-05e969\""
Oct 30 18:17:46.510819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.510778    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f5f089c9-dockercfg-vmbgz\""
Oct 30 18:17:46.530047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.530015    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:46.619157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.619123    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqt6\" (UniqueName: \"kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6\") pod \"payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.619320 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.619258    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.697110 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.697084    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"]
Oct 30 18:17:46.699012 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:17:46.698982    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21547e45_8608_47f5_b0f1_5aec9b96124f.slice/crio-65811314a3fe9deefcc8ef52c371852cda12f4a11b8a341377a8cc7e1f5e42ad WatchSource:0}: Error finding container 65811314a3fe9deefcc8ef52c371852cda12f4a11b8a341377a8cc7e1f5e42ad: Status 404 returned error can't find the container with id 65811314a3fe9deefcc8ef52c371852cda12f4a11b8a341377a8cc7e1f5e42ad
Oct 30 18:17:46.700524 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.700507    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:17:46.720424 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.720394    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqt6\" (UniqueName: \"kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6\") pod \"payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.720613 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.720466    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.721153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.721132    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.743510 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.743483    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqt6\" (UniqueName: \"kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6\") pod \"payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:46.755713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.755676    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5" event={"ID":"21547e45-8608-47f5-b0f1-5aec9b96124f","Type":"ContainerStarted","Data":"65811314a3fe9deefcc8ef52c371852cda12f4a11b8a341377a8cc7e1f5e42ad"}
Oct 30 18:17:46.778060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:46.778017    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:47.083118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.083094    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"]
Oct 30 18:17:47.085175 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:17:47.085144    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c6f2cc_ff08_4d52_8f81_926028d55145.slice/crio-8ab40a5f39cc0dd759b011d4fd04d47dcaffdc0d4fbbb37d27fbd3525c86baa3 WatchSource:0}: Error finding container 8ab40a5f39cc0dd759b011d4fd04d47dcaffdc0d4fbbb37d27fbd3525c86baa3: Status 404 returned error can't find the container with id 8ab40a5f39cc0dd759b011d4fd04d47dcaffdc0d4fbbb37d27fbd3525c86baa3
Oct 30 18:17:47.762522 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.762484    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5" event={"ID":"21547e45-8608-47f5-b0f1-5aec9b96124f","Type":"ContainerStarted","Data":"fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a"}
Oct 30 18:17:47.762707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.762565    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:47.764343 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.764313    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" event={"ID":"a6c6f2cc-ff08-4d52-8f81-926028d55145","Type":"ContainerStarted","Data":"a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843"}
Oct 30 18:17:47.764478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.764362    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" event={"ID":"a6c6f2cc-ff08-4d52-8f81-926028d55145","Type":"ContainerStarted","Data":"8ab40a5f39cc0dd759b011d4fd04d47dcaffdc0d4fbbb37d27fbd3525c86baa3"}
Oct 30 18:17:47.764478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.764384    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:47.803203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.803159    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5" podStartSLOduration=1.375331625 podStartE2EDuration="1.803147293s" totalImagesPullingTime="427.815668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:47.802320492 +0000 UTC m=+3873.564364305" watchObservedRunningTime="2026-10-30 18:17:47.803147293 +0000 UTC m=+3873.565191106"
Oct 30 18:17:47.847493 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.847444    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" podStartSLOduration=1.847428872 podStartE2EDuration="1.847428872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:47.846437563 +0000 UTC m=+3873.608481376" watchObservedRunningTime="2026-10-30 18:17:47.847428872 +0000 UTC m=+3873.609472686"
Oct 30 18:17:47.995942 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:47.995869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:49.580266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:49.580228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:51.403846 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:51.403814    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:53.063555 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:53.063517    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:53.776146 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:53.776120    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:17:54.684723 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:54.684611    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:55.508885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:55.508851    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:56.606006 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:56.605974    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:58.168391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:58.168360    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:58.773302 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:58.773275    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:17:59.560153 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:59.560119    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:17:59.816136 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:17:59.816050    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:00.200953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:00.200923    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:00.329751 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:00.329721    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:00.538700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:00.538585    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:01.699492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:01.699460    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:02.035085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:02.035000    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"]
Oct 30 18:18:02.046825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:02.046788    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2r5xh"]
Oct 30 18:18:02.375446 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:02.375408    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5102cfd-2685-4cb8-ba0e-4faae7dea198" path="/var/lib/kubelet/pods/b5102cfd-2685-4cb8-ba0e-4faae7dea198/volumes"
Oct 30 18:18:03.341293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:03.341259    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:04.065461 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:04.065428    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:04.978586 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:04.978547    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:06.631666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:06.631109    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:08.402108 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:08.402058    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:10.021653 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:10.021605    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:10.386229 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:10.386195    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:11.702096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:11.702042    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:13.417889 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:13.417837    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:14.975066 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:14.975035    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:15.246267 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:18:15.246179    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:18:16.606458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:16.606419    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:18.203627 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:18.203599    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:19.879770 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:19.879718    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:21.424467 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:21.424410    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:21.523881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:21.523846    5534 scope.go:122] "RemoveContainer" containerID="d7b1d66e3c8c7e9ad0bc5a48e10d3e226b696a89a5eeddcd4d24610e756c218d"
Oct 30 18:18:23.141997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:23.141962    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:24.758268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:24.758226    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:26.312992 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:26.312954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:27.977473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:27.977438    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:29.549706 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:29.549673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:31.311380 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:31.311342    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:32.943597 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:32.943561    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:34.030036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:34.030003    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"]
Oct 30 18:18:34.041969 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:34.041936    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gsblj"]
Oct 30 18:18:34.210138 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:34.210099    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:34.368542 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:34.368510    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c362a7f4-d59b-49ee-9fdf-3f37cbfafe43" path="/var/lib/kubelet/pods/c362a7f4-d59b-49ee-9fdf-3f37cbfafe43/volumes"
Oct 30 18:18:34.590197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:34.590168    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:35.594482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:35.594449    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:35.722883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:35.722844    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:36.255881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:36.255852    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:36.668456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:36.668413    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:38.063929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:38.063900    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:39.687708 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:39.687673    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:41.268038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:41.268007    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:41.844974 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:41.844926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:42.936833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:42.936799    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:44.501544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:44.501514    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:45.028441 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:45.028403    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"]
Oct 30 18:18:45.040084 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:45.040055    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l887v"]
Oct 30 18:18:45.243554 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:18:45.243520    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:18:46.066677 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:46.066628    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:46.368185 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:46.368153    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e688c04-7d56-4d31-80a0-910b5411c9d5" path="/var/lib/kubelet/pods/5e688c04-7d56-4d31-80a0-910b5411c9d5/volumes"
Oct 30 18:18:47.329819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:47.326554    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"]
Oct 30 18:18:47.428764 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:47.428724    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:47.541276 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:47.541245    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:47.805989 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:47.805954    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:48.585260 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:48.585213    5534 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-170.ec2.internal" podUID="cd29d125f430f9e0a8a7b7b6580a8dec" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:18:49.086905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.086872    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"]
Oct 30 18:18:49.087158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.087104    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5" podUID="21547e45-8608-47f5-b0f1-5aec9b96124f" containerName="maas-api" containerID="cri-o://fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a" gracePeriod=30
Oct 30 18:18:49.158008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.157977    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:49.564409 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.564386    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:18:49.644529 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.644429    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:49.647462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.647430    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls\") pod \"21547e45-8608-47f5-b0f1-5aec9b96124f\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") "
Oct 30 18:18:49.647626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.647540    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs\") pod \"21547e45-8608-47f5-b0f1-5aec9b96124f\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") "
Oct 30 18:18:49.647626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.647574    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca\") pod \"21547e45-8608-47f5-b0f1-5aec9b96124f\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") "
Oct 30 18:18:49.647626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.647619    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76\") pod \"21547e45-8608-47f5-b0f1-5aec9b96124f\" (UID: \"21547e45-8608-47f5-b0f1-5aec9b96124f\") "
Oct 30 18:18:49.648000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.647976    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca" pod "21547e45-8608-47f5-b0f1-5aec9b96124f" (UID: "21547e45-8608-47f5-b0f1-5aec9b96124f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:18:49.650376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.649957    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls" pod "21547e45-8608-47f5-b0f1-5aec9b96124f" (UID: "21547e45-8608-47f5-b0f1-5aec9b96124f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:18:49.650376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.650015    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76" pod "21547e45-8608-47f5-b0f1-5aec9b96124f" (UID: "21547e45-8608-47f5-b0f1-5aec9b96124f"). InnerVolumeSpecName "kube-api-access-pxt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:18:49.650376 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.650041    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs" pod "21547e45-8608-47f5-b0f1-5aec9b96124f" (UID: "21547e45-8608-47f5-b0f1-5aec9b96124f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:18:49.748725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.748698    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/21547e45-8608-47f5-b0f1-5aec9b96124f-kube-api-access-pxt76\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:18:49.748725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.748724    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:18:49.748725 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.748734    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21547e45-8608-47f5-b0f1-5aec9b96124f-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:18:49.748963 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.748744    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21547e45-8608-47f5-b0f1-5aec9b96124f-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:18:49.756993 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:49.756955    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:50.100786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.100755    5534 generic.go:356] "Generic (PLEG): container finished" podID="21547e45-8608-47f5-b0f1-5aec9b96124f" containerID="fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a" exitCode=0
Oct 30 18:18:50.100941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.100802    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"
Oct 30 18:18:50.100941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.100839    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5" event={"ID":"21547e45-8608-47f5-b0f1-5aec9b96124f","Type":"ContainerDied","Data":"fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a"}
Oct 30 18:18:50.100941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.100894    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5" event={"ID":"21547e45-8608-47f5-b0f1-5aec9b96124f","Type":"ContainerDied","Data":"65811314a3fe9deefcc8ef52c371852cda12f4a11b8a341377a8cc7e1f5e42ad"}
Oct 30 18:18:50.100941 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.100917    5534 scope.go:122] "RemoveContainer" containerID="fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a"
Oct 30 18:18:50.110373 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.110357    5534 scope.go:122] "RemoveContainer" containerID="fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a"
Oct 30 18:18:50.110663 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:18:50.110620    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a\": container with ID starting with fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a not found: ID does not exist" containerID="fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a"
Oct 30 18:18:50.110754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.110671    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a"} err="failed to get container status \"fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a\": rpc error: code = NotFound desc = could not find container \"fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a\": container with ID starting with fe1006a70ce11d588e4f525df2291047457878017e3ece0f116270b6be49f03a not found: ID does not exist"
Oct 30 18:18:50.149145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.149120    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"]
Oct 30 18:18:50.160603 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.160581    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-05e969-7b5fb4c6f-nn5f5"]
Oct 30 18:18:50.291585 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.291550    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:50.369148 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:50.369051    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21547e45-8608-47f5-b0f1-5aec9b96124f" path="/var/lib/kubelet/pods/21547e45-8608-47f5-b0f1-5aec9b96124f/volumes"
Oct 30 18:18:52.327600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:52.327553    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerName="payload-pre-processing" containerID="cri-o://a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843" gracePeriod=25
Oct 30 18:18:55.632082 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:55.632041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:55.967335 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:55.967247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:56.082855 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:56.082811    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:56.333444 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:56.333351    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:18:57.554822 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.554783    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"]
Oct 30 18:18:57.566544 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.566515    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h6t65"]
Oct 30 18:18:57.603418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.603392    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"]
Oct 30 18:18:57.603812 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.603799    5534 cpu_manager.go:414] "containerMap: removing container" podUID="21547e45-8608-47f5-b0f1-5aec9b96124f" containerName="maas-api"
Oct 30 18:18:57.603868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.603813    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21547e45-8608-47f5-b0f1-5aec9b96124f" containerName="maas-api"
Oct 30 18:18:57.603914 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.603881    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="21547e45-8608-47f5-b0f1-5aec9b96124f" containerName="maas-api"
Oct 30 18:18:57.608124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.608099    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.622291 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.622270    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:18:57.622403 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.622307    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:18:57.622497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.622479    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:18:57.625146 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.625126    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:18:57.627468 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.627445    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"]
Oct 30 18:18:57.728804 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.728769    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.729038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.728877    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrns5\" (UniqueName: \"kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.729038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.728939    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.829870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.829775    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.829870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.829828    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrns5\" (UniqueName: \"kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.830145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.829872    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.830547 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.830521    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.832315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.832298    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.852261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.852235    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrns5\" (UniqueName: \"kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5\") pod \"maas-api-revoke-sub-3d4156f1-z4nwt\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:57.921364 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:57.921343    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:18:58.224288 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:58.224083    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"]
Oct 30 18:18:58.226700 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:18:58.226674    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b744e_04bd_463d_93e6_e67cb6d90ef6.slice/crio-e9fdc651e3a75c69d5ef669569c3647aa9b81c55e7042cfe40f9d597ec879587 WatchSource:0}: Error finding container e9fdc651e3a75c69d5ef669569c3647aa9b81c55e7042cfe40f9d597ec879587: Status 404 returned error can't find the container with id e9fdc651e3a75c69d5ef669569c3647aa9b81c55e7042cfe40f9d597ec879587
Oct 30 18:18:58.368967 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:58.368934    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa43c06-06f0-4a10-bd23-8790990fb139" path="/var/lib/kubelet/pods/2aa43c06-06f0-4a10-bd23-8790990fb139/volumes"
Oct 30 18:18:58.771497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:58.771457    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:18:59.148534 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:59.148498    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt" event={"ID":"417b744e-04bd-463d-93e6-e67cb6d90ef6","Type":"ContainerStarted","Data":"a6d4193377ab2dd9bea2f745aad076274f1c75a9862920a991a4f86ef7c13b47"}
Oct 30 18:18:59.148736 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:59.148548    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt" event={"ID":"417b744e-04bd-463d-93e6-e67cb6d90ef6","Type":"ContainerStarted","Data":"e9fdc651e3a75c69d5ef669569c3647aa9b81c55e7042cfe40f9d597ec879587"}
Oct 30 18:18:59.188006 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:18:59.187960    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt" podStartSLOduration=2.187945749 podStartE2EDuration="2.187945749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:59.186329313 +0000 UTC m=+3944.948373162" watchObservedRunningTime="2026-10-30 18:18:59.187945749 +0000 UTC m=+3944.949989562"
Oct 30 18:19:00.155244 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:00.155208    5534 generic.go:356] "Generic (PLEG): container finished" podID="417b744e-04bd-463d-93e6-e67cb6d90ef6" containerID="a6d4193377ab2dd9bea2f745aad076274f1c75a9862920a991a4f86ef7c13b47" exitCode=0
Oct 30 18:19:00.155786 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:00.155252    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt" event={"ID":"417b744e-04bd-463d-93e6-e67cb6d90ef6","Type":"ContainerDied","Data":"a6d4193377ab2dd9bea2f745aad076274f1c75a9862920a991a4f86ef7c13b47"}
Oct 30 18:19:01.298011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.297983    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:19:01.467072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.466996    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca\") pod \"417b744e-04bd-463d-93e6-e67cb6d90ef6\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") "
Oct 30 18:19:01.467072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.467070    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca\") pod \"417b744e-04bd-463d-93e6-e67cb6d90ef6\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") "
Oct 30 18:19:01.467328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.467100    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5\" (UniqueName: \"kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5\") pod \"417b744e-04bd-463d-93e6-e67cb6d90ef6\" (UID: \"417b744e-04bd-463d-93e6-e67cb6d90ef6\") "
Oct 30 18:19:01.467412 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.467323    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca" pod "417b744e-04bd-463d-93e6-e67cb6d90ef6" (UID: "417b744e-04bd-463d-93e6-e67cb6d90ef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:19:01.469297 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.469274    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca" pod "417b744e-04bd-463d-93e6-e67cb6d90ef6" (UID: "417b744e-04bd-463d-93e6-e67cb6d90ef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:19:01.469426 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.469323    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5" pod "417b744e-04bd-463d-93e6-e67cb6d90ef6" (UID: "417b744e-04bd-463d-93e6-e67cb6d90ef6"). InnerVolumeSpecName "kube-api-access-lrns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:19:01.491074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.491051    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:19:01.568089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.568063    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrns5\" (UniqueName: \"kubernetes.io/projected/417b744e-04bd-463d-93e6-e67cb6d90ef6-kube-api-access-lrns5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:01.568089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.568090    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:01.568252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:01.568105    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b744e-04bd-463d-93e6-e67cb6d90ef6-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:02.166403 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:02.166372    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"
Oct 30 18:19:02.166574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:02.166412    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt" event={"ID":"417b744e-04bd-463d-93e6-e67cb6d90ef6","Type":"ContainerDied","Data":"e9fdc651e3a75c69d5ef669569c3647aa9b81c55e7042cfe40f9d597ec879587"}
Oct 30 18:19:02.166574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:02.166462    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9fdc651e3a75c69d5ef669569c3647aa9b81c55e7042cfe40f9d597ec879587"
Oct 30 18:19:08.771591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:08.771555    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:19:10.572453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:10.572375    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:19:12.575363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.575340    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:19:12.629875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.629843    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6\" (UniqueName: \"kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6\") pod \"a6c6f2cc-ff08-4d52-8f81-926028d55145\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") "
Oct 30 18:19:12.630374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.629927    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume\") pod \"a6c6f2cc-ff08-4d52-8f81-926028d55145\" (UID: \"a6c6f2cc-ff08-4d52-8f81-926028d55145\") "
Oct 30 18:19:12.630374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.630252    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume" pod "a6c6f2cc-ff08-4d52-8f81-926028d55145" (UID: "a6c6f2cc-ff08-4d52-8f81-926028d55145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:19:12.632098 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.632075    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6" pod "a6c6f2cc-ff08-4d52-8f81-926028d55145" (UID: "a6c6f2cc-ff08-4d52-8f81-926028d55145"). InnerVolumeSpecName "kube-api-access-bzqt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:19:12.731121 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.731086    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzqt6\" (UniqueName: \"kubernetes.io/projected/a6c6f2cc-ff08-4d52-8f81-926028d55145-kube-api-access-bzqt6\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:12.731121 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:12.731118    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c6f2cc-ff08-4d52-8f81-926028d55145-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:13.250665 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.250612    5534 generic.go:356] "Generic (PLEG): container finished" podID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerID="a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843" exitCode=0
Oct 30 18:19:13.250883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.250688    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"
Oct 30 18:19:13.250883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.250696    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" event={"ID":"a6c6f2cc-ff08-4d52-8f81-926028d55145","Type":"ContainerDied","Data":"a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843"}
Oct 30 18:19:13.250883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.250755    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th" event={"ID":"a6c6f2cc-ff08-4d52-8f81-926028d55145","Type":"ContainerDied","Data":"8ab40a5f39cc0dd759b011d4fd04d47dcaffdc0d4fbbb37d27fbd3525c86baa3"}
Oct 30 18:19:13.250883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.250780    5534 scope.go:122] "RemoveContainer" containerID="a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843"
Oct 30 18:19:13.261454 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.261427    5534 scope.go:122] "RemoveContainer" containerID="a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843"
Oct 30 18:19:13.261730 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:19:13.261707    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843\": container with ID starting with a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843 not found: ID does not exist" containerID="a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843"
Oct 30 18:19:13.261826 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.261739    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843"} err="failed to get container status \"a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843\": rpc error: code = NotFound desc = could not find container \"a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843\": container with ID starting with a0399092ca8864a468434bb0a8a43d0d5653e290bd33fcc7198eb3154743b843 not found: ID does not exist"
Oct 30 18:19:13.299496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.299473    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"]
Oct 30 18:19:13.311081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:13.311059    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-05e969-5f56bth6th"]
Oct 30 18:19:14.368000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:14.367968    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" path="/var/lib/kubelet/pods/a6c6f2cc-ff08-4d52-8f81-926028d55145/volumes"
Oct 30 18:19:15.244249 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:19:15.244210    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:19:18.242868 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:18.242836    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:19:19.400929 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:19.400893    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:19:21.543787 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:21.543759    5534 scope.go:122] "RemoveContainer" containerID="1af86b7c0fcbbc277f083ca0d0310a95fbfe3a23fcedfc366bea6dd6d9c4cfa5"
Oct 30 18:19:21.841351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:21.841314    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"]
Oct 30 18:19:21.841606 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:21.841567    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585" podUID="3418abd6-5f92-4c85-a9e1-b63c95219594" containerName="manager" containerID="cri-o://e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa" gracePeriod=10
Oct 30 18:19:22.098174 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.098116    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 18:19:22.221280 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.221253    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f\" (UniqueName: \"kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f\") pod \"3418abd6-5f92-4c85-a9e1-b63c95219594\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") "
Oct 30 18:19:22.221491 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.221390    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume\") pod \"3418abd6-5f92-4c85-a9e1-b63c95219594\" (UID: \"3418abd6-5f92-4c85-a9e1-b63c95219594\") "
Oct 30 18:19:22.221818 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.221792    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume" pod "3418abd6-5f92-4c85-a9e1-b63c95219594" (UID: "3418abd6-5f92-4c85-a9e1-b63c95219594"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:19:22.223667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.223607    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f" pod "3418abd6-5f92-4c85-a9e1-b63c95219594" (UID: "3418abd6-5f92-4c85-a9e1-b63c95219594"). InnerVolumeSpecName "kube-api-access-jsw6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:19:22.298108 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.298078    5534 generic.go:356] "Generic (PLEG): container finished" podID="3418abd6-5f92-4c85-a9e1-b63c95219594" containerID="e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa" exitCode=0
Oct 30 18:19:22.298266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.298116    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"
Oct 30 18:19:22.298266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.298167    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585" event={"ID":"3418abd6-5f92-4c85-a9e1-b63c95219594","Type":"ContainerDied","Data":"e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa"}
Oct 30 18:19:22.298266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.298227    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585" event={"ID":"3418abd6-5f92-4c85-a9e1-b63c95219594","Type":"ContainerDied","Data":"de5402c528766269a8419abc5b5abe2b98a69b200fcc93de3cb5b03a6d02a93b"}
Oct 30 18:19:22.298266 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.298257    5534 scope.go:122] "RemoveContainer" containerID="e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa"
Oct 30 18:19:22.308593 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.308578    5534 scope.go:122] "RemoveContainer" containerID="e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa"
Oct 30 18:19:22.308893 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:19:22.308872    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa\": container with ID starting with e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa not found: ID does not exist" containerID="e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa"
Oct 30 18:19:22.308983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.308903    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa"} err="failed to get container status \"e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa\": rpc error: code = NotFound desc = could not find container \"e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa\": container with ID starting with e2a8f7f88323632582d22bc39f241d7be7ad71280e79ebfd46d161bdb7a6ccaa not found: ID does not exist"
Oct 30 18:19:22.323158 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.323139    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsw6f\" (UniqueName: \"kubernetes.io/projected/3418abd6-5f92-4c85-a9e1-b63c95219594-kube-api-access-jsw6f\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:22.323235 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.323160    5534 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3418abd6-5f92-4c85-a9e1-b63c95219594-extensions-socket-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:19:22.349672 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.349593    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"]
Oct 30 18:19:22.352028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.352001    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-g7585"]
Oct 30 18:19:22.368595 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:22.368567    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3418abd6-5f92-4c85-a9e1-b63c95219594" path="/var/lib/kubelet/pods/3418abd6-5f92-4c85-a9e1-b63c95219594/volumes"
Oct 30 18:19:23.027713 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:23.027681    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"]
Oct 30 18:19:23.039390 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:23.039291    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98lhw"]
Oct 30 18:19:24.369775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:24.369738    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97424be-bd9e-4d2a-a380-2315f84ef6c2" path="/var/lib/kubelet/pods/a97424be-bd9e-4d2a-a380-2315f84ef6c2/volumes"
Oct 30 18:19:25.225784 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:25.225751    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:19:33.028169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:33.028138    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"]
Oct 30 18:19:33.040058 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:33.040033    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l62z7"]
Oct 30 18:19:34.352829 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:34.352797    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:19:34.368257 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:34.368222    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25679fd6-61f0-464b-8537-3a8249c4bf76" path="/var/lib/kubelet/pods/25679fd6-61f0-464b-8537-3a8249c4bf76/volumes"
Oct 30 18:19:45.244184 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:19:45.244145    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:19:55.768946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:19:55.768905    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:20:10.793732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:10.793697    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:20:15.243038 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:20:15.242997    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:20:21.581994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:21.581960    5534 scope.go:122] "RemoveContainer" containerID="3991bc945751d090feb3ca8d8a252e21bda8789e6ba9931b5163b8f0b22fd740"
Oct 30 18:20:21.591112 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:21.591084    5534 scope.go:122] "RemoveContainer" containerID="1999c8c5b44b6d704b9ddaaebcc8641ea84a83f9500afaecb76583b5c9643e7c"
Oct 30 18:20:21.599450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:21.599430    5534 scope.go:122] "RemoveContainer" containerID="1d35b9f44baceba6e283b401106b154a5532bfbfe604645d37d7d80f3b544cdb"
Oct 30 18:20:25.102427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102390    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"]
Oct 30 18:20:25.102895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102822    5534 cpu_manager.go:414] "containerMap: removing container" podUID="417b744e-04bd-463d-93e6-e67cb6d90ef6" containerName="revoke-keys"
Oct 30 18:20:25.102895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102835    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417b744e-04bd-463d-93e6-e67cb6d90ef6" containerName="revoke-keys"
Oct 30 18:20:25.102895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102847    5534 cpu_manager.go:414] "containerMap: removing container" podUID="3418abd6-5f92-4c85-a9e1-b63c95219594" containerName="manager"
Oct 30 18:20:25.102895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102852    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3418abd6-5f92-4c85-a9e1-b63c95219594" containerName="manager"
Oct 30 18:20:25.102895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102866    5534 cpu_manager.go:414] "containerMap: removing container" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerName="payload-pre-processing"
Oct 30 18:20:25.102895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102871    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerName="payload-pre-processing"
Oct 30 18:20:25.103134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102925    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c6f2cc-ff08-4d52-8f81-926028d55145" containerName="payload-pre-processing"
Oct 30 18:20:25.103134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102939    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="417b744e-04bd-463d-93e6-e67cb6d90ef6" containerName="revoke-keys"
Oct 30 18:20:25.103134 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.102947    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="3418abd6-5f92-4c85-a9e1-b63c95219594" containerName="manager"
Oct 30 18:20:25.107157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.107141    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.120096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.120068    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"]
Oct 30 18:20:25.129203 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.129181    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4ptvx\""
Oct 30 18:20:25.227395 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.227349    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfhs\" (UniqueName: \"kubernetes.io/projected/83df9c92-08ea-41d1-9259-c555d8438658-kube-api-access-djfhs\") pod \"kuadrant-operator-controller-manager-57bb9bfff-v2kxp\" (UID: \"83df9c92-08ea-41d1-9259-c555d8438658\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.227607 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.227507    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83df9c92-08ea-41d1-9259-c555d8438658-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-v2kxp\" (UID: \"83df9c92-08ea-41d1-9259-c555d8438658\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.329085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.329040    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83df9c92-08ea-41d1-9259-c555d8438658-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-v2kxp\" (UID: \"83df9c92-08ea-41d1-9259-c555d8438658\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.329307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.329135    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfhs\" (UniqueName: \"kubernetes.io/projected/83df9c92-08ea-41d1-9259-c555d8438658-kube-api-access-djfhs\") pod \"kuadrant-operator-controller-manager-57bb9bfff-v2kxp\" (UID: \"83df9c92-08ea-41d1-9259-c555d8438658\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.329515 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.329490    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83df9c92-08ea-41d1-9259-c555d8438658-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-v2kxp\" (UID: \"83df9c92-08ea-41d1-9259-c555d8438658\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.352407 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.352375    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfhs\" (UniqueName: \"kubernetes.io/projected/83df9c92-08ea-41d1-9259-c555d8438658-kube-api-access-djfhs\") pod \"kuadrant-operator-controller-manager-57bb9bfff-v2kxp\" (UID: \"83df9c92-08ea-41d1-9259-c555d8438658\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.420198 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.420113    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:25.747124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:25.747097    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"]
Oct 30 18:20:25.749705 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:20:25.749669    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83df9c92_08ea_41d1_9259_c555d8438658.slice/crio-6bf4c5838b2b6efaa37971d7c988655df33f17a424a54ee0e30f4bfc74edffda WatchSource:0}: Error finding container 6bf4c5838b2b6efaa37971d7c988655df33f17a424a54ee0e30f4bfc74edffda: Status 404 returned error can't find the container with id 6bf4c5838b2b6efaa37971d7c988655df33f17a424a54ee0e30f4bfc74edffda
Oct 30 18:20:26.629359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:26.629317    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp" event={"ID":"83df9c92-08ea-41d1-9259-c555d8438658","Type":"ContainerStarted","Data":"5b685753311a00b36cb58a7ec4cb69a6fdfa561bccfc19c5f9c9f01cc983bfa0"}
Oct 30 18:20:26.629832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:26.629381    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp" event={"ID":"83df9c92-08ea-41d1-9259-c555d8438658","Type":"ContainerStarted","Data":"6bf4c5838b2b6efaa37971d7c988655df33f17a424a54ee0e30f4bfc74edffda"}
Oct 30 18:20:26.629832 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:26.629444    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:26.699303 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:26.699251    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp" podStartSLOduration=1.699238238 podStartE2EDuration="1.699238238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:26.68697428 +0000 UTC m=+4032.449018092" watchObservedRunningTime="2026-10-30 18:20:26.699238238 +0000 UTC m=+4032.461282051"
Oct 30 18:20:36.027230 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:36.027193    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"]
Oct 30 18:20:36.039038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:36.039009    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nhxrp"]
Oct 30 18:20:36.368166 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:36.368135    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50120b3f-5e49-4d0d-a34c-81835091dd34" path="/var/lib/kubelet/pods/50120b3f-5e49-4d0d-a34c-81835091dd34/volumes"
Oct 30 18:20:37.637398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:37.637371    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-v2kxp"
Oct 30 18:20:41.872254 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:41.872215    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:20:45.361039 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:20:45.360999    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:20:47.408792 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.408762    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"]
Oct 30 18:20:47.412989 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.412970    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.427131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.427106    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:20:47.438995 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.438916    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:20:47.439143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.439101    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:20:47.440952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.440933    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"]
Oct 30 18:20:47.482551 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.482514    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9bl\" (UniqueName: \"kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.482789 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.482660    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.482789 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.482741    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.527208 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.527171    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:20:47.583804 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.583767    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9bl\" (UniqueName: \"kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.584030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.583857    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.584030 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.583926    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.584588 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.584567    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.586687 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.586617    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.606399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.606372    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9bl\" (UniqueName: \"kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl\") pod \"maas-api-revoke-sub-d576e68b-6xntb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:47.726895 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:47.726866    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:48.025707 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:48.025652    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"]
Oct 30 18:20:48.028460 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:20:48.028423    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf850109d_f69d_462c_96b3_67895277c8cb.slice/crio-1506f2fc8bbb6ca7e4698c52f3c7ab953570b25614142645b99aeb605b722b7c WatchSource:0}: Error finding container 1506f2fc8bbb6ca7e4698c52f3c7ab953570b25614142645b99aeb605b722b7c: Status 404 returned error can't find the container with id 1506f2fc8bbb6ca7e4698c52f3c7ab953570b25614142645b99aeb605b722b7c
Oct 30 18:20:48.773398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:48.773297    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb" event={"ID":"f850109d-f69d-462c-96b3-67895277c8cb","Type":"ContainerStarted","Data":"e7c961a3bb2f50b375321e5a5cd3705b8ef9e2da076bb55c6a7ce54941a6285d"}
Oct 30 18:20:48.773398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:48.773357    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb" event={"ID":"f850109d-f69d-462c-96b3-67895277c8cb","Type":"ContainerStarted","Data":"1506f2fc8bbb6ca7e4698c52f3c7ab953570b25614142645b99aeb605b722b7c"}
Oct 30 18:20:48.812876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:48.812825    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb" podStartSLOduration=1.812810786 podStartE2EDuration="1.812810786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:48.811333815 +0000 UTC m=+4054.573377629" watchObservedRunningTime="2026-10-30 18:20:48.812810786 +0000 UTC m=+4054.574854599"
Oct 30 18:20:49.786292 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:49.786247    5534 generic.go:356] "Generic (PLEG): container finished" podID="f850109d-f69d-462c-96b3-67895277c8cb" containerID="e7c961a3bb2f50b375321e5a5cd3705b8ef9e2da076bb55c6a7ce54941a6285d" exitCode=0
Oct 30 18:20:49.786732 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:49.786322    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb" event={"ID":"f850109d-f69d-462c-96b3-67895277c8cb","Type":"ContainerDied","Data":"e7c961a3bb2f50b375321e5a5cd3705b8ef9e2da076bb55c6a7ce54941a6285d"}
Oct 30 18:20:50.937255 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:50.937230    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:51.016137 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.016098    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca\") pod \"f850109d-f69d-462c-96b3-67895277c8cb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") "
Oct 30 18:20:51.016348 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.016244    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl\" (UniqueName: \"kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl\") pod \"f850109d-f69d-462c-96b3-67895277c8cb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") "
Oct 30 18:20:51.016348 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.016302    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca\") pod \"f850109d-f69d-462c-96b3-67895277c8cb\" (UID: \"f850109d-f69d-462c-96b3-67895277c8cb\") "
Oct 30 18:20:51.016584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.016550    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca" pod "f850109d-f69d-462c-96b3-67895277c8cb" (UID: "f850109d-f69d-462c-96b3-67895277c8cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:20:51.016791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.016772    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:20:51.018594 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.018570    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca" pod "f850109d-f69d-462c-96b3-67895277c8cb" (UID: "f850109d-f69d-462c-96b3-67895277c8cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:20:51.018712 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.018596    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl" pod "f850109d-f69d-462c-96b3-67895277c8cb" (UID: "f850109d-f69d-462c-96b3-67895277c8cb"). InnerVolumeSpecName "kube-api-access-bq9bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:20:51.117877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.117844    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9bl\" (UniqueName: \"kubernetes.io/projected/f850109d-f69d-462c-96b3-67895277c8cb-kube-api-access-bq9bl\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:20:51.117877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.117872    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f850109d-f69d-462c-96b3-67895277c8cb-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:20:51.797162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.797120    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb" event={"ID":"f850109d-f69d-462c-96b3-67895277c8cb","Type":"ContainerDied","Data":"1506f2fc8bbb6ca7e4698c52f3c7ab953570b25614142645b99aeb605b722b7c"}
Oct 30 18:20:51.797391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.797184    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1506f2fc8bbb6ca7e4698c52f3c7ab953570b25614142645b99aeb605b722b7c"
Oct 30 18:20:51.797391 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:51.797136    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"
Oct 30 18:20:55.899313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:20:55.899222    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:06.028410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:06.028376    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"]
Oct 30 18:21:06.039957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:06.039923    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jn6lq"]
Oct 30 18:21:06.368614 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:06.368575    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575fffa3-f8cc-4f28-8770-ac9912224d40" path="/var/lib/kubelet/pods/575fffa3-f8cc-4f28-8770-ac9912224d40/volumes"
Oct 30 18:21:10.830227 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:10.830172    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:14.503432 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.503399    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"]
Oct 30 18:21:14.503892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.503814    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f850109d-f69d-462c-96b3-67895277c8cb" containerName="revoke-keys"
Oct 30 18:21:14.503892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.503825    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f850109d-f69d-462c-96b3-67895277c8cb" containerName="revoke-keys"
Oct 30 18:21:14.503892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.503890    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f850109d-f69d-462c-96b3-67895277c8cb" containerName="revoke-keys"
Oct 30 18:21:14.507912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.507895    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.522116 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.522097    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:21:14.533691 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.533667    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:21:14.533837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.533721    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:21:14.533837 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.533720    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:21:14.535303 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.535283    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"]
Oct 30 18:21:14.590872 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.590837    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.591057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.590946    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.591057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.590982    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66xh\" (UniqueName: \"kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.692028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.691987    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.692028 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.692032    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66xh\" (UniqueName: \"kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.692285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.692143    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.692793 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.692775    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.694449 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.694427    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.714819 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.714779    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66xh\" (UniqueName: \"kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh\") pod \"maas-api-revoke-sub-d2bc93bc-xx86j\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:14.820264 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:14.820175    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:15.119061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:15.119032    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"]
Oct 30 18:21:15.121341 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:21:15.121311    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24aa0ca7_a48a_4e47_a2ad_3438d30e7298.slice/crio-6405aa0ac61a350f46dc4d22b50d189316ae7478c530ca51cb5d978a6cd887d7 WatchSource:0}: Error finding container 6405aa0ac61a350f46dc4d22b50d189316ae7478c530ca51cb5d978a6cd887d7: Status 404 returned error can't find the container with id 6405aa0ac61a350f46dc4d22b50d189316ae7478c530ca51cb5d978a6cd887d7
Oct 30 18:21:15.242578 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:21:15.242540    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:21:15.269241 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:15.269216    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:15.933290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:15.933257    5534 generic.go:356] "Generic (PLEG): container finished" podID="24aa0ca7-a48a-4e47-a2ad-3438d30e7298" containerID="cfb970327d455c8d337ec38b357677ffe5893f73e38943c26f0af01853ffe678" exitCode=0
Oct 30 18:21:15.933686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:15.933349    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j" event={"ID":"24aa0ca7-a48a-4e47-a2ad-3438d30e7298","Type":"ContainerDied","Data":"cfb970327d455c8d337ec38b357677ffe5893f73e38943c26f0af01853ffe678"}
Oct 30 18:21:15.933686 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:15.933402    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j" event={"ID":"24aa0ca7-a48a-4e47-a2ad-3438d30e7298","Type":"ContainerStarted","Data":"6405aa0ac61a350f46dc4d22b50d189316ae7478c530ca51cb5d978a6cd887d7"}
Oct 30 18:21:16.123298 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:16.123243    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:17.076372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.076348    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:17.218610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.218522    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca\") pod \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") "
Oct 30 18:21:17.218610 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.218573    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca\") pod \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") "
Oct 30 18:21:17.218886 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.218618    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh\" (UniqueName: \"kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh\") pod \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\" (UID: \"24aa0ca7-a48a-4e47-a2ad-3438d30e7298\") "
Oct 30 18:21:17.219019 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.218992    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca" pod "24aa0ca7-a48a-4e47-a2ad-3438d30e7298" (UID: "24aa0ca7-a48a-4e47-a2ad-3438d30e7298"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:21:17.219131 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.219075    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:21:17.220959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.220933    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca" pod "24aa0ca7-a48a-4e47-a2ad-3438d30e7298" (UID: "24aa0ca7-a48a-4e47-a2ad-3438d30e7298"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:21:17.220959 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.220952    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh" pod "24aa0ca7-a48a-4e47-a2ad-3438d30e7298" (UID: "24aa0ca7-a48a-4e47-a2ad-3438d30e7298"). InnerVolumeSpecName "kube-api-access-g66xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:21:17.319918 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.319882    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:21:17.319918 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.319916    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66xh\" (UniqueName: \"kubernetes.io/projected/24aa0ca7-a48a-4e47-a2ad-3438d30e7298-kube-api-access-g66xh\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:21:17.947101 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.947065    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"
Oct 30 18:21:17.947290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.947110    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j" event={"ID":"24aa0ca7-a48a-4e47-a2ad-3438d30e7298","Type":"ContainerDied","Data":"6405aa0ac61a350f46dc4d22b50d189316ae7478c530ca51cb5d978a6cd887d7"}
Oct 30 18:21:17.947290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:17.947197    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6405aa0ac61a350f46dc4d22b50d189316ae7478c530ca51cb5d978a6cd887d7"
Oct 30 18:21:21.633436 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:21.633402    5534 scope.go:122] "RemoveContainer" containerID="758a1a6d64bdd0d76bf8c701b3ec4280a2bd706d656920a1389b081207581295"
Oct 30 18:21:23.036287 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:23.036258    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"]
Oct 30 18:21:23.047823 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:23.047786    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clk8x"]
Oct 30 18:21:24.367753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:24.367710    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f847b91-8daa-4dc4-b93e-023abb5969a6" path="/var/lib/kubelet/pods/5f847b91-8daa-4dc4-b93e-023abb5969a6/volumes"
Oct 30 18:21:39.465877 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:39.465840    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:43.406398 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:43.406365    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:43.902480 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:43.902443    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:44.020958 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:44.020921    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:45.244031 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:21:45.243997    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:21:46.818533 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:46.818490    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:46.931078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:46.931047    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:47.879240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:47.879205    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:48.019634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:48.019550    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:48.903133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:48.903100    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:49.035331 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.035302    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:49.374875 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.374840    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"]
Oct 30 18:21:49.375324 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.375309    5534 cpu_manager.go:414] "containerMap: removing container" podUID="24aa0ca7-a48a-4e47-a2ad-3438d30e7298" containerName="revoke-keys"
Oct 30 18:21:49.375373 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.375326    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24aa0ca7-a48a-4e47-a2ad-3438d30e7298" containerName="revoke-keys"
Oct 30 18:21:49.375415 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.375398    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="24aa0ca7-a48a-4e47-a2ad-3438d30e7298" containerName="revoke-keys"
Oct 30 18:21:49.380074 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.380053    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.406382 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.406355    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-67382b-serving-cert\""
Oct 30 18:21:49.406634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.406381    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:21:49.406634 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.406442    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s8xph\""
Oct 30 18:21:49.406871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.406483    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:21:49.406871 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.406770    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:21:49.408393 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.408372    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"]
Oct 30 18:21:49.467101 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.467073    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:49.538791 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.538757    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9z9j\" (UniqueName: \"kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.539000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.538833    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.539000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.538966    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.539105 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.539013    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.592059 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.592030    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:49.640332 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.640254    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9z9j\" (UniqueName: \"kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.640560 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.640333    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.640560 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.640377    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.640560 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.640401    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.641193 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.641117    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.642950 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.642929    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.643101 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.642978    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.664081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.664051    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9z9j\" (UniqueName: \"kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j\") pod \"maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.694971 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.694935    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:49.715572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.715537    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:49.750968 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.750935    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"]
Oct 30 18:21:49.755981 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.755867    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:49.775813 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.774954    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ec894879-dockercfg-ldcl6\""
Oct 30 18:21:49.777259 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.777157    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"]
Oct 30 18:21:49.808983 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.808922    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-67382b\""
Oct 30 18:21:49.842124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.842092    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:49.842124 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.842134    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxtx\" (UniqueName: \"kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx\") pod \"payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:49.943600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.943501    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:49.943600 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.943541    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxtx\" (UniqueName: \"kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx\") pod \"payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:49.944410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.944388    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:49.965991 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:49.965963    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxtx\" (UniqueName: \"kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx\") pod \"payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:50.003169 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:50.003146    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"]
Oct 30 18:21:50.005529 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:21:50.005500    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5219b0_1f3b_4d0a_b45c_137484dcfd52.slice/crio-6374b2134cd2c03fcf815350bfc3658cd1aae0002f8aa199d1a271716c1f3f73 WatchSource:0}: Error finding container 6374b2134cd2c03fcf815350bfc3658cd1aae0002f8aa199d1a271716c1f3f73: Status 404 returned error can't find the container with id 6374b2134cd2c03fcf815350bfc3658cd1aae0002f8aa199d1a271716c1f3f73
Oct 30 18:21:50.085396 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:50.085366    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:50.107303 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:50.107271    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt" event={"ID":"4c5219b0-1f3b-4d0a-b45c-137484dcfd52","Type":"ContainerStarted","Data":"6374b2134cd2c03fcf815350bfc3658cd1aae0002f8aa199d1a271716c1f3f73"}
Oct 30 18:21:50.207224 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:50.207133    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:50.390080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:50.390045    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:50.390212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:50.390113    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"]
Oct 30 18:21:50.397183 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:21:50.397163    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b53ab2d_b872_4d87_bf25_8e50a7f6a6c2.slice/crio-cf5428861d33eaeefa9eed759bd44f64129b990dd6c023e3336339aeb4d66331 WatchSource:0}: Error finding container cf5428861d33eaeefa9eed759bd44f64129b990dd6c023e3336339aeb4d66331: Status 404 returned error can't find the container with id cf5428861d33eaeefa9eed759bd44f64129b990dd6c023e3336339aeb4d66331
Oct 30 18:21:51.115313 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.115279    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" event={"ID":"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2","Type":"ContainerStarted","Data":"eedf202737558aff53d532482785c491a1c1d3833e67e03531a294e1e043abe0"}
Oct 30 18:21:51.115801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.115335    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" event={"ID":"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2","Type":"ContainerStarted","Data":"cf5428861d33eaeefa9eed759bd44f64129b990dd6c023e3336339aeb4d66331"}
Oct 30 18:21:51.115801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.115363    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:21:51.117223 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.117195    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt" event={"ID":"4c5219b0-1f3b-4d0a-b45c-137484dcfd52","Type":"ContainerStarted","Data":"3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f"}
Oct 30 18:21:51.117351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.117333    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:51.155277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.155228    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" podStartSLOduration=2.155214147 podStartE2EDuration="2.155214147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:51.153250576 +0000 UTC m=+4116.915294389" watchObservedRunningTime="2026-10-30 18:21:51.155214147 +0000 UTC m=+4116.917257994"
Oct 30 18:21:51.195062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.195010    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt" podStartSLOduration=1.822089804 podStartE2EDuration="2.194995691s" totalImagesPullingTime="372.905887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:51.193496508 +0000 UTC m=+4116.955540320" watchObservedRunningTime="2026-10-30 18:21:51.194995691 +0000 UTC m=+4116.957039553"
Oct 30 18:21:51.245007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:51.244972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:52.995459 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:52.995391    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:54.797160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:54.797114    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:56.555381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:56.555331    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:57.128057 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:57.128030    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:21:58.084318 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:58.084255    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:21:59.791999 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:21:59.791966    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:01.432492 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:01.432458    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:02.125568 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:02.125538    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:22:02.804372 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:02.804333    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:03.130775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:03.130726    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:03.421256 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:03.421146    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:03.642590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:03.642531    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:04.915076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:04.915041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:05.869582 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:05.869544    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:06.692538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:06.692507    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:06.907206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:06.907170    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:08.274262 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:08.274228    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:10.101362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:10.101327    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:10.938485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:10.938450    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:11.741152 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:11.741110    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:13.382284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:13.382247    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:15.009211 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:15.009177    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:15.243723 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:22:15.243685    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:22:16.652157 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:16.652125    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:18.322453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:18.322422    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:19.855040 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:19.855003    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:21.519787 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:21.519755    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:21.655135 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:21.655098    5534 scope.go:122] "RemoveContainer" containerID="673c2637ffad6e69af2c94ccec2d6916a2f7985e5908dc2e02766ac6659bbe19"
Oct 30 18:22:21.663898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:21.663880    5534 scope.go:122] "RemoveContainer" containerID="66f9d96016fa5631c5f6a30d884f54741338c4106d7bb4a2aba1547cbcffaca9"
Oct 30 18:22:21.671675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:21.671655    5534 scope.go:122] "RemoveContainer" containerID="a36ae3c00e0fde055b6417b39d5730353785a1b33c891b5748dc48390e5ce305"
Oct 30 18:22:23.094916 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:23.094883    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:24.812385 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:24.812297    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:24.987502 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:24.987470    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:26.377965 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:26.377926    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:28.082410 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:28.082369    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:29.647096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:29.647064    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:31.265290 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:31.265255    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:33.004615 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:33.004580    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:34.609801 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:34.609766    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:36.282111 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:36.282076    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:37.815374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:37.815340    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:39.396023 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:39.395989    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:41.096905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:41.096869    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:42.641838 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:42.641801    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:44.202505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:44.202471    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:45.243882 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:22:45.243850    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:22:45.692982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:45.692945    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:45.913735 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:45.913702    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:47.779976 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:47.779940    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:50.719731 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:50.719698    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:56.371035 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:56.371003    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"]
Oct 30 18:22:56.480430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:56.480392    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:56.597316 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:56.597285    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:56.708389 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:56.708299    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:58.539496 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:58.539466    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"]
Oct 30 18:22:58.540023 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:58.539673    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt" podUID="4c5219b0-1f3b-4d0a-b45c-137484dcfd52" containerName="maas-api" containerID="cri-o://3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f" gracePeriod=30
Oct 30 18:22:58.609439 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:58.609400    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:59.025805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.025454    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:22:59.144444 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.144412    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls\") pod \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") "
Oct 30 18:22:59.144699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.144622    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j\" (UniqueName: \"kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j\") pod \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") "
Oct 30 18:22:59.144800 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.144729    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca\") pod \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") "
Oct 30 18:22:59.144800 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.144793    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs\") pod \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\" (UID: \"4c5219b0-1f3b-4d0a-b45c-137484dcfd52\") "
Oct 30 18:22:59.145211 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.145136    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca" pod "4c5219b0-1f3b-4d0a-b45c-137484dcfd52" (UID: "4c5219b0-1f3b-4d0a-b45c-137484dcfd52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:22:59.146857 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.146826    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls" pod "4c5219b0-1f3b-4d0a-b45c-137484dcfd52" (UID: "4c5219b0-1f3b-4d0a-b45c-137484dcfd52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:22:59.147011 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.146989    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j" pod "4c5219b0-1f3b-4d0a-b45c-137484dcfd52" (UID: "4c5219b0-1f3b-4d0a-b45c-137484dcfd52"). InnerVolumeSpecName "kube-api-access-f9z9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:22:59.147251 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.147230    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs" pod "4c5219b0-1f3b-4d0a-b45c-137484dcfd52" (UID: "4c5219b0-1f3b-4d0a-b45c-137484dcfd52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:22:59.229226 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.229185    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:59.246456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.246421    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9z9j\" (UniqueName: \"kubernetes.io/projected/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-kube-api-access-f9z9j\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:22:59.246456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.246450    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:22:59.246669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.246466    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:22:59.246669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.246481    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5219b0-1f3b-4d0a-b45c-137484dcfd52-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:22:59.345083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.345041    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:22:59.467246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.467158    5534 generic.go:356] "Generic (PLEG): container finished" podID="4c5219b0-1f3b-4d0a-b45c-137484dcfd52" containerID="3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f" exitCode=0
Oct 30 18:22:59.467246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.467204    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"
Oct 30 18:22:59.467246 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.467225    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt" event={"ID":"4c5219b0-1f3b-4d0a-b45c-137484dcfd52","Type":"ContainerDied","Data":"3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f"}
Oct 30 18:22:59.467497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.467270    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt" event={"ID":"4c5219b0-1f3b-4d0a-b45c-137484dcfd52","Type":"ContainerDied","Data":"6374b2134cd2c03fcf815350bfc3658cd1aae0002f8aa199d1a271716c1f3f73"}
Oct 30 18:22:59.467497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.467292    5534 scope.go:122] "RemoveContainer" containerID="3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f"
Oct 30 18:22:59.476845 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.476827    5534 scope.go:122] "RemoveContainer" containerID="3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f"
Oct 30 18:22:59.477112 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:22:59.477094    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f\": container with ID starting with 3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f not found: ID does not exist" containerID="3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f"
Oct 30 18:22:59.477168 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.477121    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f"} err="failed to get container status \"3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f\": rpc error: code = NotFound desc = could not find container \"3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f\": container with ID starting with 3d5dabc0c8c9d40288a939236582dce93482f19126e92fd26c1fc066adf8a31f not found: ID does not exist"
Oct 30 18:22:59.515883 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.515855    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"]
Oct 30 18:22:59.527330 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.527292    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-67382b-574d9b6477-j2hmt"]
Oct 30 18:22:59.910699 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:22:59.910631    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:00.367374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:00.367342    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5219b0-1f3b-4d0a-b45c-137484dcfd52" path="/var/lib/kubelet/pods/4c5219b0-1f3b-4d0a-b45c-137484dcfd52/volumes"
Oct 30 18:23:01.089745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:01.089710    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:01.371951 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:01.371847    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerName="payload-pre-processing" containerID="cri-o://eedf202737558aff53d532482785c491a1c1d3833e67e03531a294e1e043abe0" gracePeriod=25
Oct 30 18:23:02.123408 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:02.123367    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:23:05.785584 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:05.785548    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:05.908911 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:05.908875    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:06.142659 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:06.142612    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:11.069876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:11.069839    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:12.123748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:12.123711    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:23:15.243318 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:23:15.243280    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:23:21.325026 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.324988    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:21.584160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.584078    5534 generic.go:356] "Generic (PLEG): container finished" podID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerID="eedf202737558aff53d532482785c491a1c1d3833e67e03531a294e1e043abe0" exitCode=0
Oct 30 18:23:21.584160 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.584144    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" event={"ID":"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2","Type":"ContainerDied","Data":"eedf202737558aff53d532482785c491a1c1d3833e67e03531a294e1e043abe0"}
Oct 30 18:23:21.624430 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.624408    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:23:21.661535 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.661512    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx\" (UniqueName: \"kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx\") pod \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") "
Oct 30 18:23:21.661704 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.661621    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume\") pod \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\" (UID: \"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2\") "
Oct 30 18:23:21.662016 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.661997    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume" pod "1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" (UID: "1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:23:21.663919 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.663896    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx" pod "1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" (UID: "1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2"). InnerVolumeSpecName "kube-api-access-xtxtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:23:21.697746 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.697721    5534 scope.go:122] "RemoveContainer" containerID="1cf2521d2b67780da6626f7d69e559ed4847c6c4202bc810a8f0b7f2dcd266c9"
Oct 30 18:23:21.708454 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.708418    5534 scope.go:122] "RemoveContainer" containerID="215e3cb1cc714fea04e51498fd25f9335ea0e1c7d68f115b91c707df6f1d0612"
Oct 30 18:23:21.718141 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.718120    5534 scope.go:122] "RemoveContainer" containerID="f002d0087bfc4a0227da792e46f81cb8de52ef310ef507caba8effb645353ee2"
Oct 30 18:23:21.762688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.762631    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtxtx\" (UniqueName: \"kubernetes.io/projected/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-kube-api-access-xtxtx\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:23:21.762688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:21.762686    5534 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2-plugins-config-volume\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:23:22.590216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:22.590178    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl" event={"ID":"1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2","Type":"ContainerDied","Data":"cf5428861d33eaeefa9eed759bd44f64129b990dd6c023e3336339aeb4d66331"}
Oct 30 18:23:22.590760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:22.590239    5534 scope.go:122] "RemoveContainer" containerID="eedf202737558aff53d532482785c491a1c1d3833e67e03531a294e1e043abe0"
Oct 30 18:23:22.590760 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:22.590189    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"
Oct 30 18:23:22.639273 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:22.639166    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"]
Oct 30 18:23:22.650996 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:22.650954    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-67382b-6b6bctvgfl"]
Oct 30 18:23:24.387192 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:24.387145    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" path="/var/lib/kubelet/pods/1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2/volumes"
Oct 30 18:23:28.017749 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:28.017703    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:28.281805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:28.281723    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:33.855754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:33.855716    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:42.459478 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:42.459437    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:44.696469 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:44.696423    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:23:45.257084 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:23:45.257002    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:23:47.930949 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:47.930907    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8ttz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:23:47.931591 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:47.931554    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:23:48.138798 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:48.138744    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j4pdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:23:48.139154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:23:48.139107    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:24:02.028458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:02.028376    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"]
Oct 30 18:24:02.039876 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:02.039849    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z4nwt"]
Oct 30 18:24:02.367654 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:02.367609    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417b744e-04bd-463d-93e6-e67cb6d90ef6" path="/var/lib/kubelet/pods/417b744e-04bd-463d-93e6-e67cb6d90ef6/volumes"
Oct 30 18:24:09.842411 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842377    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"]
Oct 30 18:24:09.843080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842819    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerName="payload-pre-processing"
Oct 30 18:24:09.843080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842832    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerName="payload-pre-processing"
Oct 30 18:24:09.843080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842848    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4c5219b0-1f3b-4d0a-b45c-137484dcfd52" containerName="maas-api"
Oct 30 18:24:09.843080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842853    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5219b0-1f3b-4d0a-b45c-137484dcfd52" containerName="maas-api"
Oct 30 18:24:09.843080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842919    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b53ab2d-b872-4d87-bf25-8e50a7f6a6c2" containerName="payload-pre-processing"
Oct 30 18:24:09.843080 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.842929    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5219b0-1f3b-4d0a-b45c-137484dcfd52" containerName="maas-api"
Oct 30 18:24:09.846197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.846178    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:09.860545 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.860524    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:24:09.868090 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.868068    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:24:09.869756 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.869736    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"]
Oct 30 18:24:09.904975 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.904914    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:09.905140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.905051    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzb5\" (UniqueName: \"kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:09.905140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.905084    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:09.940997 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.940974    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:24:09.941123 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:09.940997    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:24:10.005802 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.005777    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzb5\" (UniqueName: \"kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.005946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.005812    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.005946 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.005915    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.006560 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.006531    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.008701 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.008676    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.028573 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.028545    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzb5\" (UniqueName: \"kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5\") pod \"maas-api-revoke-sub-452b6d7b-xgxsl\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.158807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.158720    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:10.462532 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.462503    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"]
Oct 30 18:24:10.465282 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:24:10.465250    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6127da7_5bf9_4cf7_ab2c_00f60915ea76.slice/crio-48af456042ed9783407aa16e512cf4b4ca19caaae5551daeddd2af57f9b03775 WatchSource:0}: Error finding container 48af456042ed9783407aa16e512cf4b4ca19caaae5551daeddd2af57f9b03775: Status 404 returned error can't find the container with id 48af456042ed9783407aa16e512cf4b4ca19caaae5551daeddd2af57f9b03775
Oct 30 18:24:10.937752 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.937712    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl" event={"ID":"d6127da7-5bf9-4cf7-ab2c-00f60915ea76","Type":"ContainerStarted","Data":"9b35a7198725b17efe706588552a4c4e81ccb65f7989d1d49348f3fc76e802d4"}
Oct 30 18:24:10.938232 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.937772    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl" event={"ID":"d6127da7-5bf9-4cf7-ab2c-00f60915ea76","Type":"ContainerStarted","Data":"48af456042ed9783407aa16e512cf4b4ca19caaae5551daeddd2af57f9b03775"}
Oct 30 18:24:10.977294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:10.977239    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl" podStartSLOduration=1.977225805 podStartE2EDuration="1.977225805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:10.97572401 +0000 UTC m=+4256.737767823" watchObservedRunningTime="2026-10-30 18:24:10.977225805 +0000 UTC m=+4256.739269624"
Oct 30 18:24:11.947268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:11.947229    5534 generic.go:356] "Generic (PLEG): container finished" podID="d6127da7-5bf9-4cf7-ab2c-00f60915ea76" containerID="9b35a7198725b17efe706588552a4c4e81ccb65f7989d1d49348f3fc76e802d4" exitCode=0
Oct 30 18:24:11.947853 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:11.947316    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl" event={"ID":"d6127da7-5bf9-4cf7-ab2c-00f60915ea76","Type":"ContainerDied","Data":"9b35a7198725b17efe706588552a4c4e81ccb65f7989d1d49348f3fc76e802d4"}
Oct 30 18:24:13.745085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.744804    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:13.839014 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.838978    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5\" (UniqueName: \"kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5\") pod \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") "
Oct 30 18:24:13.839743 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.839718    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca\") pod \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") "
Oct 30 18:24:13.839994 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.839975    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca\") pod \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\" (UID: \"d6127da7-5bf9-4cf7-ab2c-00f60915ea76\") "
Oct 30 18:24:13.843927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.843808    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca" pod "d6127da7-5bf9-4cf7-ab2c-00f60915ea76" (UID: "d6127da7-5bf9-4cf7-ab2c-00f60915ea76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:24:13.851904 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.844365    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:24:13.852199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.849115    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5" pod "d6127da7-5bf9-4cf7-ab2c-00f60915ea76" (UID: "d6127da7-5bf9-4cf7-ab2c-00f60915ea76"). InnerVolumeSpecName "kube-api-access-nmzb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:24:13.900668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.900009    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca" pod "d6127da7-5bf9-4cf7-ab2c-00f60915ea76" (UID: "d6127da7-5bf9-4cf7-ab2c-00f60915ea76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:24:13.955429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.955274    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:24:13.955429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.955312    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzb5\" (UniqueName: \"kubernetes.io/projected/d6127da7-5bf9-4cf7-ab2c-00f60915ea76-kube-api-access-nmzb5\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:24:13.995666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.992756    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"
Oct 30 18:24:13.996199 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.992670    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl" event={"ID":"d6127da7-5bf9-4cf7-ab2c-00f60915ea76","Type":"ContainerDied","Data":"48af456042ed9783407aa16e512cf4b4ca19caaae5551daeddd2af57f9b03775"}
Oct 30 18:24:13.996368 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:13.996351    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48af456042ed9783407aa16e512cf4b4ca19caaae5551daeddd2af57f9b03775"
Oct 30 18:24:15.250986 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:24:15.250930    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:24:15.291848 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:15.291810    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:24:15.292045 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:15.291878    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:24:21.108048 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:21.108007    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:24:21.226912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:21.226879    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:24:24.027049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:24.027010    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:24:27.681414 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.681365    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"]
Oct 30 18:24:27.682038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.682018    5534 cpu_manager.go:414] "containerMap: removing container" podUID="d6127da7-5bf9-4cf7-ab2c-00f60915ea76" containerName="revoke-keys"
Oct 30 18:24:27.682115 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.682040    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6127da7-5bf9-4cf7-ab2c-00f60915ea76" containerName="revoke-keys"
Oct 30 18:24:27.682193 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.682154    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6127da7-5bf9-4cf7-ab2c-00f60915ea76" containerName="revoke-keys"
Oct 30 18:24:27.691753 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.691720    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.714258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.714226    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"]
Oct 30 18:24:27.727805 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.727768    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:24:27.739378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.739154    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:24:27.739378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.739227    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:24:27.739378 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.739318    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:24:27.791776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.791743    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.792047 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.792024    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wmwn\" (UniqueName: \"kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.792268 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.792248    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.893299 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.893262    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.894217 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.894184    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.894450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.894427    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wmwn\" (UniqueName: \"kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.895035 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.894348    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.897844 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.897816    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:27.919921 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:27.919889    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wmwn\" (UniqueName: \"kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn\") pod \"maas-api-revoke-sub-3e31a506-hdtbq\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:28.047570 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:28.047465    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:28.385912 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:28.385383    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"]
Oct 30 18:24:28.389552 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:24:28.389526    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1569d523_d744_4319_be22_62c2e3843364.slice/crio-3ec0418030db4a9844bd913189e6af99a89f20b9f49d6e75d0d620f0f1efb7d7 WatchSource:0}: Error finding container 3ec0418030db4a9844bd913189e6af99a89f20b9f49d6e75d0d620f0f1efb7d7: Status 404 returned error can't find the container with id 3ec0418030db4a9844bd913189e6af99a89f20b9f49d6e75d0d620f0f1efb7d7
Oct 30 18:24:29.115667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:29.115385    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq" event={"ID":"1569d523-d744-4319-be22-62c2e3843364","Type":"ContainerStarted","Data":"d219163e05e682004e93dcda60487001a0e5d38bc26bd107f800279faf3af067"}
Oct 30 18:24:29.115667 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:29.115470    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq" event={"ID":"1569d523-d744-4319-be22-62c2e3843364","Type":"ContainerStarted","Data":"3ec0418030db4a9844bd913189e6af99a89f20b9f49d6e75d0d620f0f1efb7d7"}
Oct 30 18:24:29.160794 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:29.156976    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq" podStartSLOduration=2.156958303 podStartE2EDuration="2.156958303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:29.155058588 +0000 UTC m=+4274.917102431" watchObservedRunningTime="2026-10-30 18:24:29.156958303 +0000 UTC m=+4274.919002115"
Oct 30 18:24:30.136669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:30.136195    5534 generic.go:356] "Generic (PLEG): container finished" podID="1569d523-d744-4319-be22-62c2e3843364" containerID="d219163e05e682004e93dcda60487001a0e5d38bc26bd107f800279faf3af067" exitCode=0
Oct 30 18:24:30.136669 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:30.136349    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq" event={"ID":"1569d523-d744-4319-be22-62c2e3843364","Type":"ContainerDied","Data":"d219163e05e682004e93dcda60487001a0e5d38bc26bd107f800279faf3af067"}
Oct 30 18:24:30.351668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:30.350771    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-92jb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:24:30.351668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:30.350864    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:24:31.276173 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.276148    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:31.408572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.408496    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca\") pod \"1569d523-d744-4319-be22-62c2e3843364\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") "
Oct 30 18:24:31.408744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.408669    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn\" (UniqueName: \"kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn\") pod \"1569d523-d744-4319-be22-62c2e3843364\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") "
Oct 30 18:24:31.408744 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.408706    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca\") pod \"1569d523-d744-4319-be22-62c2e3843364\" (UID: \"1569d523-d744-4319-be22-62c2e3843364\") "
Oct 30 18:24:31.408900 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.408874    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca" pod "1569d523-d744-4319-be22-62c2e3843364" (UID: "1569d523-d744-4319-be22-62c2e3843364"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:24:31.409071 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.409051    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1569d523-d744-4319-be22-62c2e3843364-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:24:31.411017 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.410985    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca" pod "1569d523-d744-4319-be22-62c2e3843364" (UID: "1569d523-d744-4319-be22-62c2e3843364"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:24:31.411161 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.411065    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn" pod "1569d523-d744-4319-be22-62c2e3843364" (UID: "1569d523-d744-4319-be22-62c2e3843364"). InnerVolumeSpecName "kube-api-access-8wmwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:24:31.510065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.510033    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wmwn\" (UniqueName: \"kubernetes.io/projected/1569d523-d744-4319-be22-62c2e3843364-kube-api-access-8wmwn\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:24:31.510065 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:31.510061    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1569d523-d744-4319-be22-62c2e3843364-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:24:32.148508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:32.148475    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"
Oct 30 18:24:32.148757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:32.148473    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq" event={"ID":"1569d523-d744-4319-be22-62c2e3843364","Type":"ContainerDied","Data":"3ec0418030db4a9844bd913189e6af99a89f20b9f49d6e75d0d620f0f1efb7d7"}
Oct 30 18:24:32.148757 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:24:32.148608    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec0418030db4a9844bd913189e6af99a89f20b9f49d6e75d0d620f0f1efb7d7"
Oct 30 18:24:45.242964 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:24:45.242929    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:25:15.245522 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:25:15.245483    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:25:15.971559 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:15.971525    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:25:16.805720 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:16.805680    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:25:18.385898 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.385868    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"]
Oct 30 18:25:18.386483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.386463    5534 cpu_manager.go:414] "containerMap: removing container" podUID="1569d523-d744-4319-be22-62c2e3843364" containerName="revoke-keys"
Oct 30 18:25:18.386574 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.386487    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1569d523-d744-4319-be22-62c2e3843364" containerName="revoke-keys"
Oct 30 18:25:18.386632 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.386596    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="1569d523-d744-4319-be22-62c2e3843364" containerName="revoke-keys"
Oct 30 18:25:18.389783 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.389764    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.416096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.415841    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\""
Oct 30 18:25:18.416096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.415970    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:25:18.416096 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.416070    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:25:18.417688 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.417659    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"]
Oct 30 18:25:18.480338 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.480299    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.480538 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.480355    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.480623 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.480540    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5k2w\" (UniqueName: \"kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.505099 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.505074    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:25:18.581062 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.581033    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5k2w\" (UniqueName: \"kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.581263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.581092    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.581263 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.581128    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.581948 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.581924    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.583624 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.583603    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.604033 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.604006    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5k2w\" (UniqueName: \"kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w\") pod \"maas-api-revoke-sub-08378079-bmflz\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.703010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.702921    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:18.999577 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:18.999548    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"]
Oct 30 18:25:19.001885 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:25:19.001855    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d54532_697d_4b3d_bf8a_2936968fbfcc.slice/crio-20d7ba798d09603036a901426e6861c16e054153c9cbe79ba0f7938dacc56e21 WatchSource:0}: Error finding container 20d7ba798d09603036a901426e6861c16e054153c9cbe79ba0f7938dacc56e21: Status 404 returned error can't find the container with id 20d7ba798d09603036a901426e6861c16e054153c9cbe79ba0f7938dacc56e21
Oct 30 18:25:19.493215 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:19.493173    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz" event={"ID":"96d54532-697d-4b3d-bf8a-2936968fbfcc","Type":"ContainerStarted","Data":"deeb8dbb9268919ee6e08447d38453ff2f2aec4e501055f0ad7d1d959ba0e10e"}
Oct 30 18:25:19.493604 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:19.493239    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz" event={"ID":"96d54532-697d-4b3d-bf8a-2936968fbfcc","Type":"ContainerStarted","Data":"20d7ba798d09603036a901426e6861c16e054153c9cbe79ba0f7938dacc56e21"}
Oct 30 18:25:19.533543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:19.533441    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz" podStartSLOduration=1.53342781 podStartE2EDuration="1.53342781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:19.532080253 +0000 UTC m=+4325.294124066" watchObservedRunningTime="2026-10-30 18:25:19.53342781 +0000 UTC m=+4325.295471623"
Oct 30 18:25:20.499543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:20.499507    5534 generic.go:356] "Generic (PLEG): container finished" podID="96d54532-697d-4b3d-bf8a-2936968fbfcc" containerID="deeb8dbb9268919ee6e08447d38453ff2f2aec4e501055f0ad7d1d959ba0e10e" exitCode=0
Oct 30 18:25:20.499968 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:20.499582    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz" event={"ID":"96d54532-697d-4b3d-bf8a-2936968fbfcc","Type":"ContainerDied","Data":"deeb8dbb9268919ee6e08447d38453ff2f2aec4e501055f0ad7d1d959ba0e10e"}
Oct 30 18:25:21.321061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.321013    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:25:21.649165 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.649136    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:21.713671 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.713604    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca\") pod \"96d54532-697d-4b3d-bf8a-2936968fbfcc\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") "
Oct 30 18:25:21.713870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.713702    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca\") pod \"96d54532-697d-4b3d-bf8a-2936968fbfcc\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") "
Oct 30 18:25:21.713870 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.713777    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w\" (UniqueName: \"kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w\") pod \"96d54532-697d-4b3d-bf8a-2936968fbfcc\" (UID: \"96d54532-697d-4b3d-bf8a-2936968fbfcc\") "
Oct 30 18:25:21.714218 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.714189    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca" pod "96d54532-697d-4b3d-bf8a-2936968fbfcc" (UID: "96d54532-697d-4b3d-bf8a-2936968fbfcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:25:21.716061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.716033    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca" pod "96d54532-697d-4b3d-bf8a-2936968fbfcc" (UID: "96d54532-697d-4b3d-bf8a-2936968fbfcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:25:21.716061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.716045    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w" pod "96d54532-697d-4b3d-bf8a-2936968fbfcc" (UID: "96d54532-697d-4b3d-bf8a-2936968fbfcc"). InnerVolumeSpecName "kube-api-access-w5k2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:25:21.761811 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.761777    5534 scope.go:122] "RemoveContainer" containerID="a6d4193377ab2dd9bea2f745aad076274f1c75a9862920a991a4f86ef7c13b47"
Oct 30 18:25:21.815475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.815442    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-xks-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:21.815475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.815474    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d54532-697d-4b3d-bf8a-2936968fbfcc-maas-api-openshift-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:21.815734 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:21.815486    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5k2w\" (UniqueName: \"kubernetes.io/projected/96d54532-697d-4b3d-bf8a-2936968fbfcc-kube-api-access-w5k2w\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:22.510252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:22.510227    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"
Oct 30 18:25:22.510252 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:22.510236    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz" event={"ID":"96d54532-697d-4b3d-bf8a-2936968fbfcc","Type":"ContainerDied","Data":"20d7ba798d09603036a901426e6861c16e054153c9cbe79ba0f7938dacc56e21"}
Oct 30 18:25:22.510475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:22.510279    5534 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d7ba798d09603036a901426e6861c16e054153c9cbe79ba0f7938dacc56e21"
Oct 30 18:25:30.667691 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.667591    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"]
Oct 30 18:25:30.668315 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.668294    5534 cpu_manager.go:414] "containerMap: removing container" podUID="96d54532-697d-4b3d-bf8a-2936968fbfcc" containerName="revoke-keys"
Oct 30 18:25:30.668427 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.668317    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d54532-697d-4b3d-bf8a-2936968fbfcc" containerName="revoke-keys"
Oct 30 18:25:30.668489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.668443    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d54532-697d-4b3d-bf8a-2936968fbfcc" containerName="revoke-keys"
Oct 30 18:25:30.671828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.671803    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.697697 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.697631    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:25:30.697866 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.697715    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:25:30.699961 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.699937    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"]
Oct 30 18:25:30.735479 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.735447    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 18:25:30.735692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.735476    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:25:30.735692 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.735498    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s8xph\""
Oct 30 18:25:30.805549 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.805508    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.805803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.805558    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvlp\" (UniqueName: \"kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.805803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.805718    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.805803 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.805743    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.906481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.906447    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.906481 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.906482    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvlp\" (UniqueName: \"kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.906778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.906509    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.906778 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.906532    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.907388 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.907362    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.909204 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.909180    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.909354 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.909332    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.929240 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.929184    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvlp\" (UniqueName: \"kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp\") pod \"maas-api-5c48877c75-qr2v6\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:30.986489 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:30.986454    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:31.287375 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:31.287346    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"]
Oct 30 18:25:31.290418 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:25:31.290388    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4688e222_f7ce_4b0d_89ea_b4f88f7fa2b0.slice/crio-92cdb0e864bc0f39193e5178c255c9959269191e89dc57c57b8e39b70f076caf WatchSource:0}: Error finding container 92cdb0e864bc0f39193e5178c255c9959269191e89dc57c57b8e39b70f076caf: Status 404 returned error can't find the container with id 92cdb0e864bc0f39193e5178c255c9959269191e89dc57c57b8e39b70f076caf
Oct 30 18:25:31.292088 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:31.292069    5534 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:25:31.553497 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:31.553397    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6" event={"ID":"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0","Type":"ContainerStarted","Data":"92cdb0e864bc0f39193e5178c255c9959269191e89dc57c57b8e39b70f076caf"}
Oct 30 18:25:32.560658 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:32.560606    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6" event={"ID":"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0","Type":"ContainerStarted","Data":"24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801"}
Oct 30 18:25:32.561078 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:32.560776    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:32.604679 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:32.602589    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6" podStartSLOduration=2.142640206 podStartE2EDuration="2.602571014s" totalImagesPullingTime="459.930808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:32.599933902 +0000 UTC m=+4338.361977734" watchObservedRunningTime="2026-10-30 18:25:32.602571014 +0000 UTC m=+4338.364614830"
Oct 30 18:25:38.570939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.570897    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:38.827418 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.827345    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"]
Oct 30 18:25:38.831145 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.831126    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.852927 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.852900    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"]
Oct 30 18:25:38.884029 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.883995    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2xs\" (UniqueName: \"kubernetes.io/projected/eba65813-5f94-492c-a072-0e36496d48b8-kube-api-access-8d2xs\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.884221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.884112    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba65813-5f94-492c-a072-0e36496d48b8-maas-api-tls\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.884221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.884153    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65813-5f94-492c-a072-0e36496d48b8-openshift-service-ca\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.884221 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.884174    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba65813-5f94-492c-a072-0e36496d48b8-metrics-certs\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.984754 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.984717    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba65813-5f94-492c-a072-0e36496d48b8-maas-api-tls\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.984945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.984766    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65813-5f94-492c-a072-0e36496d48b8-openshift-service-ca\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.984945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.984793    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba65813-5f94-492c-a072-0e36496d48b8-metrics-certs\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.984945 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.984917    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2xs\" (UniqueName: \"kubernetes.io/projected/eba65813-5f94-492c-a072-0e36496d48b8-kube-api-access-8d2xs\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.985477 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.985449    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65813-5f94-492c-a072-0e36496d48b8-openshift-service-ca\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.987543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.987512    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba65813-5f94-492c-a072-0e36496d48b8-maas-api-tls\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:38.987543 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:38.987535    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba65813-5f94-492c-a072-0e36496d48b8-metrics-certs\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:39.007462 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:39.007436    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2xs\" (UniqueName: \"kubernetes.io/projected/eba65813-5f94-492c-a072-0e36496d48b8-kube-api-access-8d2xs\") pod \"maas-api-595cc648c-2v4qb\" (UID: \"eba65813-5f94-492c-a072-0e36496d48b8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:39.144775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:39.144740    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:39.446118 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:39.446096    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"]
Oct 30 18:25:39.448786 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:25:39.448758    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba65813_5f94_492c_a072_0e36496d48b8.slice/crio-85ed8a759d652f9f0778adb99eeacd005d1b81584ee3ac8cab3d73c63b47deca WatchSource:0}: Error finding container 85ed8a759d652f9f0778adb99eeacd005d1b81584ee3ac8cab3d73c63b47deca: Status 404 returned error can't find the container with id 85ed8a759d652f9f0778adb99eeacd005d1b81584ee3ac8cab3d73c63b47deca
Oct 30 18:25:39.593092 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:39.593054    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb" event={"ID":"eba65813-5f94-492c-a072-0e36496d48b8","Type":"ContainerStarted","Data":"85ed8a759d652f9f0778adb99eeacd005d1b81584ee3ac8cab3d73c63b47deca"}
Oct 30 18:25:40.602797 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:40.602748    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb" event={"ID":"eba65813-5f94-492c-a072-0e36496d48b8","Type":"ContainerStarted","Data":"c7815727cb30f10ce0619d4cb98fafc0b6c700be68d7b2ad4ba2e3dac49aeaa4"}
Oct 30 18:25:40.603626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:40.603602    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:40.645422 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:40.643781    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb" podStartSLOduration=2.29966942 podStartE2EDuration="2.643763219s" totalImagesPullingTime="344.093799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:40.643536015 +0000 UTC m=+4346.405579832" watchObservedRunningTime="2026-10-30 18:25:40.643763219 +0000 UTC m=+4346.405807034"
Oct 30 18:25:45.243907 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:25:45.243868    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:25:47.643499 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:47.643471    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-2v4qb"
Oct 30 18:25:47.716768 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:47.716722    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"]
Oct 30 18:25:47.717025 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:47.716959    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6" podUID="4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" containerName="maas-api" containerID="cri-o://24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801" gracePeriod=30
Oct 30 18:25:47.981264 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:47.981237    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:48.050139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.050110    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls\") pod \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") "
Oct 30 18:25:48.050327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.050169    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca\") pod \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") "
Oct 30 18:25:48.050327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.050216    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp\" (UniqueName: \"kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp\") pod \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") "
Oct 30 18:25:48.050327 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.050283    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs\") pod \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\" (UID: \"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0\") "
Oct 30 18:25:48.050599 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.050572    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca" pod "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" (UID: "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:25:48.050740 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.050718    5534 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-openshift-service-ca\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:48.052860 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.052837    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp" pod "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" (UID: "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0"). InnerVolumeSpecName "kube-api-access-bmvlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:25:48.053207 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.053186    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs" pod "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" (UID: "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:25:48.053301 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.053204    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls" pod "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" (UID: "4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:25:48.151928 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.151900    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmvlp\" (UniqueName: \"kubernetes.io/projected/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-kube-api-access-bmvlp\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:48.151928 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.151924    5534 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-metrics-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:48.151928 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.151936    5534 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0-maas-api-tls\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:25:48.673180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.673066    5534 generic.go:356] "Generic (PLEG): container finished" podID="4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" containerID="24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801" exitCode=0
Oct 30 18:25:48.673180 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.673124    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"
Oct 30 18:25:48.673799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.673171    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6" event={"ID":"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0","Type":"ContainerDied","Data":"24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801"}
Oct 30 18:25:48.673799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.673262    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6" event={"ID":"4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0","Type":"ContainerDied","Data":"92cdb0e864bc0f39193e5178c255c9959269191e89dc57c57b8e39b70f076caf"}
Oct 30 18:25:48.673799 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.673316    5534 scope.go:122] "RemoveContainer" containerID="24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801"
Oct 30 18:25:48.686450 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.686431    5534 scope.go:122] "RemoveContainer" containerID="24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801"
Oct 30 18:25:48.687045 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:25:48.687018    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801\": container with ID starting with 24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801 not found: ID does not exist" containerID="24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801"
Oct 30 18:25:48.687304 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.687270    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801"} err="failed to get container status \"24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801\": rpc error: code = NotFound desc = could not find container \"24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801\": container with ID starting with 24f1de8abb2fd28ce431c507b6f0fcca74868c30430ace3542e2a6999ae8f801 not found: ID does not exist"
Oct 30 18:25:48.724782 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.724733    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"]
Oct 30 18:25:48.737306 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:48.737258    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-qr2v6"]
Oct 30 18:25:49.768635 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:49.768549    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:25:50.381139 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:50.381106    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" path="/var/lib/kubelet/pods/4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0/volumes"
Oct 30 18:25:51.029666 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:51.028942    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"]
Oct 30 18:25:51.055668 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:51.052547    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6xntb"]
Oct 30 18:25:52.369001 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:25:52.368968    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f850109d-f69d-462c-96b3-67895277c8cb" path="/var/lib/kubelet/pods/f850109d-f69d-462c-96b3-67895277c8cb/volumes"
Oct 30 18:26:15.244969 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:26:15.244932    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:26:18.027337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:18.027302    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"]
Oct 30 18:26:18.039351 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:18.039324    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xx86j"]
Oct 30 18:26:18.368381 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:18.368352    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24aa0ca7-a48a-4e47-a2ad-3438d30e7298" path="/var/lib/kubelet/pods/24aa0ca7-a48a-4e47-a2ad-3438d30e7298/volumes"
Oct 30 18:26:21.452007 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:21.451972    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:26:26.331075 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:26.331038    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:26:35.096913 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:35.096879    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bc8c78ff8-kx77f_cb838fb8-f499-4b7f-a2ff-99c126affe86/authorino/0.log"
Oct 30 18:26:35.900572 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:35.900534    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:26:45.242859 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:26:45.242821    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:26:49.893172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:49.893104    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:26:55.751143 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:55.751109    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59c45c7574-t8dth_fd0270f0-f92b-4192-9b07-fb82620137f0/manager/0.log"
Oct 30 18:26:56.799466 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:26:56.799431    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bb675f547-mb27n_4eed760b-e4dc-4f18-aa57-a58ff18df081/manager/0.log"
Oct 30 18:27:08.176717 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:08.176684    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg_8a175b30-bdc4-42a1-b81b-0f5d9da955c0/copy-manifests/0.log"
Oct 30 18:27:08.230070 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:08.230037    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg_8a175b30-bdc4-42a1-b81b-0f5d9da955c0/manager/0.log"
Oct 30 18:27:08.469126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:08.469045    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-4rcsr_ccff571c-a44f-447f-a36f-fbc580cf7b0a/proxy/0.log"
Oct 30 18:27:09.296339 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:09.296309    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-2v4qb_eba65813-5f94-492c-a072-0e36496d48b8/maas-api/0.log"
Oct 30 18:27:15.243632 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:27:15.243594    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:27:19.931262 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:19.931235    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-08378079-bmflz_96d54532-697d-4b3d-bf8a-2936968fbfcc/revoke-keys/0.log"
Oct 30 18:27:20.129655 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:20.129622    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-hdtbq_1569d523-d744-4319-be22-62c2e3843364/revoke-keys/0.log"
Oct 30 18:27:20.315698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:20.315627    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-xgxsl_d6127da7-5bf9-4cf7-ab2c-00f60915ea76/revoke-keys/0.log"
Oct 30 18:27:20.817132 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:20.817068    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:27:20.930939 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:20.930901    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:27:21.567512 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:21.567476    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:27:21.794938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:21.794901    5534 scope.go:122] "RemoveContainer" containerID="e7c961a3bb2f50b375321e5a5cd3705b8ef9e2da076bb55c6a7ce54941a6285d"
Oct 30 18:27:21.803881 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:21.803859    5534 scope.go:122] "RemoveContainer" containerID="cfb970327d455c8d337ec38b357677ffe5893f73e38943c26f0af01853ffe678"
Oct 30 18:27:22.688248 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:22.688212    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bc8c78ff8-kx77f_cb838fb8-f499-4b7f-a2ff-99c126affe86/authorino/0.log"
Oct 30 18:27:23.351076 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:23.351048    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-26cq7_7ba1c859-3d6d-47d5-8ab1-72cba22b2146/kuadrant-console-plugin/0.log"
Oct 30 18:27:23.568105 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:23.568077    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-v2kxp_83df9c92-08ea-41d1-9259-c555d8438658/manager/0.log"
Oct 30 18:27:25.213473 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:25.213441    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57db9d4f85-fc579_e8a85129-d6fd-42c0-b2a3-50187e0edf0a/kube-auth-proxy/0.log"
Oct 30 18:27:25.999043 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:25.999007    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-8ttz7_f3be3002-8b68-42c9-8a9a-74e2774e277b/istio-proxy/0.log"
Oct 30 18:27:26.192475 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:26.192448    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-92jb6_cab59453-7b05-46b1-b731-cdf6fb511b38/istio-proxy/0.log"
Oct 30 18:27:35.080005 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:35.079965    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:27:36.598851 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:36.598817    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-j4pdf_5757d303-8049-4309-8652-001808326e91/istio-proxy/0.log"
Oct 30 18:27:36.801905 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:36.801877    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-lmw9v_e5a9f812-5e0d-4fb8-ab84-77d523901927/istio-proxy/0.log"
Oct 30 18:27:38.901049 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:38.901024    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj_5217b9b6-dc38-4b7e-bf14-0206e45811e6/main/0.log"
Oct 30 18:27:39.098434 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:39.098401    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-r5kxv_ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e/main/0.log"
Oct 30 18:27:39.700365 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:39.700335    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6_8cac9387-6cfc-4ded-8041-00e436212e6b/main/0.log"
Oct 30 18:27:40.076482 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:40.076403    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx_e3ec1506-9ff5-48f6-9bd7-d19a6a095892/main/0.log"
Oct 30 18:27:45.244227 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:27:45.244190    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:27:50.034216 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:50.034185    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:27:51.023632 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:27:51.023598    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bc8c78ff8-kx77f_cb838fb8-f499-4b7f-a2ff-99c126affe86/authorino/0.log"
Oct 30 18:28:15.243802 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:28:15.243766    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:28:21.671709 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:21.671671    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:28:30.255825 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:30.255743    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59c45c7574-t8dth_fd0270f0-f92b-4192-9b07-fb82620137f0/manager/0.log"
Oct 30 18:28:31.462880 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:31.462847    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:28:41.164162 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:41.164131    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bb675f547-mb27n_4eed760b-e4dc-4f18-aa57-a58ff18df081/manager/0.log"
Oct 30 18:28:42.601127 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:42.601095    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg_8a175b30-bdc4-42a1-b81b-0f5d9da955c0/manager/0.log"
Oct 30 18:28:42.643212 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:42.643189    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-p8dsg_8a175b30-bdc4-42a1-b81b-0f5d9da955c0/copy-manifests/0.log"
Oct 30 18:28:42.851622 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:42.851541    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-4rcsr_ccff571c-a44f-447f-a36f-fbc580cf7b0a/proxy/0.log"
Oct 30 18:28:43.421759 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:43.421725    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-2v4qb_eba65813-5f94-492c-a072-0e36496d48b8/maas-api/0.log"
Oct 30 18:28:44.276683 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:44.276633    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:28:45.242606 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:28:45.242569    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:28:50.153901 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:28:50.153867    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:29:01.841576 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:01.841544    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:29:04.012973 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:04.012943    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-08378079-bmflz_96d54532-697d-4b3d-bf8a-2936968fbfcc/revoke-keys/0.log"
Oct 30 18:29:04.210036 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:04.210006    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-hdtbq_1569d523-d744-4319-be22-62c2e3843364/revoke-keys/0.log"
Oct 30 18:29:04.398261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:04.398234    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-xgxsl_d6127da7-5bf9-4cf7-ab2c-00f60915ea76/revoke-keys/0.log"
Oct 30 18:29:06.171355 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:06.171324    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bc8c78ff8-kx77f_cb838fb8-f499-4b7f-a2ff-99c126affe86/authorino/0.log"
Oct 30 18:29:06.807849 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:06.807820    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-26cq7_7ba1c859-3d6d-47d5-8ab1-72cba22b2146/kuadrant-console-plugin/0.log"
Oct 30 18:29:07.015526 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:07.015496    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-v2kxp_83df9c92-08ea-41d1-9259-c555d8438658/manager/0.log"
Oct 30 18:29:14.027575 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:14.027542    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"]
Oct 30 18:29:14.039362 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:14.039333    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xgxsl"]
Oct 30 18:29:14.367662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:14.367620    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6127da7-5bf9-4cf7-ab2c-00f60915ea76" path="/var/lib/kubelet/pods/d6127da7-5bf9-4cf7-ab2c-00f60915ea76/volumes"
Oct 30 18:29:15.243938 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:29:15.243900    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:29:18.545525 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:18.545491    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57db9d4f85-fc579_e8a85129-d6fd-42c0-b2a3-50187e0edf0a/kube-auth-proxy/0.log"
Oct 30 18:29:19.393890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:19.393861    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-8ttz7_f3be3002-8b68-42c9-8a9a-74e2774e277b/istio-proxy/0.log"
Oct 30 18:29:19.599982 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:19.599951    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-92jb6_cab59453-7b05-46b1-b731-cdf6fb511b38/istio-proxy/0.log"
Oct 30 18:29:19.990293 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:19.990264    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-j4pdf_5757d303-8049-4309-8652-001808326e91/istio-proxy/0.log"
Oct 30 18:29:20.185258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:20.185226    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-lmw9v_e5a9f812-5e0d-4fb8-ab84-77d523901927/istio-proxy/0.log"
Oct 30 18:29:21.831663 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:21.831608    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:29:21.906089 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:21.906062    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9z2vj_5217b9b6-dc38-4b7e-bf14-0206e45811e6/main/0.log"
Oct 30 18:29:22.105000 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:22.104971    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-r5kxv_ea2b2b6d-7436-4a24-85ad-d81bf1b0b79e/main/0.log"
Oct 30 18:29:31.578729 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:31.578689    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:29:32.027885 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:32.027851    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"]
Oct 30 18:29:32.039995 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:32.039972    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hdtbq"]
Oct 30 18:29:32.367662 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:32.367618    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1569d523-d744-4319-be22-62c2e3843364" path="/var/lib/kubelet/pods/1569d523-d744-4319-be22-62c2e3843364/volumes"
Oct 30 18:29:42.697337 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:42.697307    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mztb6_8cac9387-6cfc-4ded-8041-00e436212e6b/main/0.log"
Oct 30 18:29:43.102085 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:43.102055    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-cbznx_e3ec1506-9ff5-48f6-9bd7-d19a6a095892/main/0.log"
Oct 30 18:29:45.244134 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:29:45.244095    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:29:50.288539 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:29:50.288505    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:30:15.244185 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:30:15.244138    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:30:21.840024 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:21.839994    5534 scope.go:122] "RemoveContainer" containerID="9b35a7198725b17efe706588552a4c4e81ccb65f7989d1d49348f3fc76e802d4"
Oct 30 18:30:21.977374 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:21.977336    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:30:22.031590 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:22.031560    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"]
Oct 30 18:30:22.043460 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:22.043433    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08378079-bmflz"]
Oct 30 18:30:22.368247 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:22.368213    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d54532-697d-4b3d-bf8a-2936968fbfcc" path="/var/lib/kubelet/pods/96d54532-697d-4b3d-bf8a-2936968fbfcc/volumes"
Oct 30 18:30:31.698458 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:31.698420    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:30:45.245260 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:30:45.245217    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:30:50.427657 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:50.427610    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:30:58.691908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:30:58.691868    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:31:15.244846 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:31:15.244801    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:31:16.456093 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:16.456059    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bb675f547-mb27n_4eed760b-e4dc-4f18-aa57-a58ff18df081/manager/0.log"
Oct 30 18:31:21.861087 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:21.861055    5534 scope.go:122] "RemoveContainer" containerID="deeb8dbb9268919ee6e08447d38453ff2f2aec4e501055f0ad7d1d959ba0e10e"
Oct 30 18:31:21.870237 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:21.870213    5534 scope.go:122] "RemoveContainer" containerID="d219163e05e682004e93dcda60487001a0e5d38bc26bd107f800279faf3af067"
Oct 30 18:31:26.813152 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:26.813061    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bb675f547-mb27n_4eed760b-e4dc-4f18-aa57-a58ff18df081/manager/0.log"
Oct 30 18:31:32.089953 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:32.089915    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:31:41.836128 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:41.836094    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:31:45.244441 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:31:45.244395    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:31:51.812431 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.812395    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"]
Oct 30 18:31:51.812937 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.812614    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" containerID="cri-o://8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481" gracePeriod=30
Oct 30 18:31:51.826046 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.826012    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"]
Oct 30 18:31:51.826261 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.826236    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy" containerID="cri-o://7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69" gracePeriod=30
Oct 30 18:31:51.839705 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.839675    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"]
Oct 30 18:31:51.840133 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.840102    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy" containerID="cri-o://7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084" gracePeriod=30
Oct 30 18:31:51.844957 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.844932    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"]
Oct 30 18:31:51.845154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:51.845134    5534 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy" containerID="cri-o://4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7" gracePeriod=30
Oct 30 18:31:56.953790 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:56.953756    5534 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lmw9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 18:31:56.954236 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:56.953822    5534 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 18:31:57.096938 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.096914    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:31:57.138457 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.138362    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:31:57.160038 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.159992    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:31:57.174554 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.174530    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 18:31:57.180485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180460    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.180637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180501    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.180637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180528    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.180637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180554    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.180637 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180582    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.180908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180766    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.180908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180829    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.180908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180849    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.180908 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180872    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180913    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180937    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180957    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.180984    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181026    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181059    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181105    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181150 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181144    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.181569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181174    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.181569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181206    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181255    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181330    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket\") pod \"f3be3002-8b68-42c9-8a9a-74e2774e277b\" (UID: \"f3be3002-8b68-42c9-8a9a-74e2774e277b\") "
Oct 30 18:31:57.181569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181371    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert\") pod \"5757d303-8049-4309-8652-001808326e91\" (UID: \"5757d303-8049-4309-8652-001808326e91\") "
Oct 30 18:31:57.181892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181513    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.181892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.181744    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.182229 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.182206    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:31:57.182334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.182205    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.182456 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.182436    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.182762 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.182726    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183113    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183135    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757d303-8049-4309-8652-001808326e91-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183151    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183168    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183182    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3be3002-8b68-42c9-8a9a-74e2774e277b-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183194    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183206    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183219    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183231    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.183267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.183243    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.184140 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.184113    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:31:57.184429 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.184407    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.184700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.184622    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "kube-api-access-8vzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.184700 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.184630    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:31:57.185301 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.185281    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.185505 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.185488    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.186258 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.186239    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token" pod "f3be3002-8b68-42c9-8a9a-74e2774e277b" (UID: "f3be3002-8b68-42c9-8a9a-74e2774e277b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.186328 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.186253    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v" pod "5757d303-8049-4309-8652-001808326e91" (UID: "5757d303-8049-4309-8652-001808326e91"). InnerVolumeSpecName "kube-api-access-l265v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.284041 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284013    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284062    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284085    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284101    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284148    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284169    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284195    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284216    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284248    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284282    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284328    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284374    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284411    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284431    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284442    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284481    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284506    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284541    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284594    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert\") pod \"cab59453-7b05-46b1-b731-cdf6fb511b38\" (UID: \"cab59453-7b05-46b1-b731-cdf6fb511b38\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284687    5534 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data\") pod \"e5a9f812-5e0d-4fb8-ab84-77d523901927\" (UID: \"e5a9f812-5e0d-4fb8-ab84-77d523901927\") "
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284690    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284744    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284777    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.284776 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.284783    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.285833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285203    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.285833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285283    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.285833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285379    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:31:57.285833 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285696    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285862    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285880    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285896    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757d303-8049-4309-8652-001808326e91-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285910    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757d303-8049-4309-8652-001808326e91-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285923    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285938    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vzcs\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-kube-api-access-8vzcs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285956    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285971    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l265v\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-kube-api-access-l265v\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285985    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757d303-8049-4309-8652-001808326e91-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.285998    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a9f812-5e0d-4fb8-ab84-77d523901927-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286011    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286024    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286037    5534 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-data\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286051    5534 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286064    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3be3002-8b68-42c9-8a9a-74e2774e277b-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.286081 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286079    5534 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-workload-certs\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.287205 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286094    5534 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-credential-socket\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.287205 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.286107    5534 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab59453-7b05-46b1-b731-cdf6fb511b38-istiod-ca-cert\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.287333 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.287218    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.287445 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.287420    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.287520 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.287461    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.287748 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.287725    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:31:57.287860 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.287844    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "kube-api-access-7bntb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.288285 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.288261    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "kube-api-access-tv8r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:31:57.288435 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.288419    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo" pod "e5a9f812-5e0d-4fb8-ab84-77d523901927" (UID: "e5a9f812-5e0d-4fb8-ab84-77d523901927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:31:57.288611 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.288593    5534 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy" pod "cab59453-7b05-46b1-b731-cdf6fb511b38" (UID: "cab59453-7b05-46b1-b731-cdf6fb511b38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387229    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387257    5534 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-token\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387265    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387273    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv8r9\" (UniqueName: \"kubernetes.io/projected/e5a9f812-5e0d-4fb8-ab84-77d523901927-kube-api-access-tv8r9\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387282    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387292    5534 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab59453-7b05-46b1-b731-cdf6fb511b38-istio-envoy\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387301    5534 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bntb\" (UniqueName: \"kubernetes.io/projected/cab59453-7b05-46b1-b731-cdf6fb511b38-kube-api-access-7bntb\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.387314 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.387312    5534 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a9f812-5e0d-4fb8-ab84-77d523901927-istio-podinfo\") on node \"ip-10-0-135-170.ec2.internal\" DevicePath \"\""
Oct 30 18:31:57.500680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.500635    5534 generic.go:356] "Generic (PLEG): container finished" podID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerID="8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481" exitCode=0
Oct 30 18:31:57.500864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.500704    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"
Oct 30 18:31:57.500864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.500740    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" event={"ID":"e5a9f812-5e0d-4fb8-ab84-77d523901927","Type":"ContainerDied","Data":"8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481"}
Oct 30 18:31:57.500864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.500802    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v" event={"ID":"e5a9f812-5e0d-4fb8-ab84-77d523901927","Type":"ContainerDied","Data":"a9c7de232679c6f6744d64d3e2f15e6b1652c49cfa3270f713bf1e18e25c30d2"}
Oct 30 18:31:57.500864 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.500831    5534 scope.go:122] "RemoveContainer" containerID="8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481"
Oct 30 18:31:57.502154 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.502131    5534 generic.go:356] "Generic (PLEG): container finished" podID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerID="4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7" exitCode=0
Oct 30 18:31:57.502284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.502178    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"
Oct 30 18:31:57.502284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.502201    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" event={"ID":"cab59453-7b05-46b1-b731-cdf6fb511b38","Type":"ContainerDied","Data":"4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7"}
Oct 30 18:31:57.502284 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.502270    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6" event={"ID":"cab59453-7b05-46b1-b731-cdf6fb511b38","Type":"ContainerDied","Data":"3f2ef714cb2650cc04e9d59e16067da8399e3b9b03004d2da0ee02e76781e607"}
Oct 30 18:31:57.503626 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.503606    5534 generic.go:356] "Generic (PLEG): container finished" podID="5757d303-8049-4309-8652-001808326e91" containerID="7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69" exitCode=0
Oct 30 18:31:57.503807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.503664    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" event={"ID":"5757d303-8049-4309-8652-001808326e91","Type":"ContainerDied","Data":"7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69"}
Oct 30 18:31:57.503807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.503691    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"
Oct 30 18:31:57.503807 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.503714    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf" event={"ID":"5757d303-8049-4309-8652-001808326e91","Type":"ContainerDied","Data":"d8ae411de7d659de669d80c4d867bb86471db7f09668021a8f856cda730ddace"}
Oct 30 18:31:57.505060 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.505043    5534 generic.go:356] "Generic (PLEG): container finished" podID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerID="7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084" exitCode=0
Oct 30 18:31:57.505179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.505076    5534 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"
Oct 30 18:31:57.505179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.505127    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" event={"ID":"f3be3002-8b68-42c9-8a9a-74e2774e277b","Type":"ContainerDied","Data":"7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084"}
Oct 30 18:31:57.505179 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.505164    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7" event={"ID":"f3be3002-8b68-42c9-8a9a-74e2774e277b","Type":"ContainerDied","Data":"3ebf85865ebdccca22d668ea3b2c143a65fadf22c584b0e81dd6c5a1947ad987"}
Oct 30 18:31:57.512063 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.512041    5534 scope.go:122] "RemoveContainer" containerID="8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481"
Oct 30 18:31:57.512379 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:31:57.512357    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481\": container with ID starting with 8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481 not found: ID does not exist" containerID="8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481"
Oct 30 18:31:57.512464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.512388    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481"} err="failed to get container status \"8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481\": rpc error: code = NotFound desc = could not find container \"8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481\": container with ID starting with 8febc0c3c418a8435f4d58ae16aef48844a681a0f39105ea8579d3c818e25481 not found: ID does not exist"
Oct 30 18:31:57.512464 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.512408    5534 scope.go:122] "RemoveContainer" containerID="4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7"
Oct 30 18:31:57.523061 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.523043    5534 scope.go:122] "RemoveContainer" containerID="4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7"
Oct 30 18:31:57.523357 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:31:57.523340    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7\": container with ID starting with 4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7 not found: ID does not exist" containerID="4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7"
Oct 30 18:31:57.523417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.523363    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7"} err="failed to get container status \"4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7\": rpc error: code = NotFound desc = could not find container \"4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7\": container with ID starting with 4ae2ddab4e17812e7740d755f76d6b610a7bdfad806695cd3fe363fb790b82a7 not found: ID does not exist"
Oct 30 18:31:57.523417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.523379    5534 scope.go:122] "RemoveContainer" containerID="7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69"
Oct 30 18:31:57.531548 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.531529    5534 scope.go:122] "RemoveContainer" containerID="7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69"
Oct 30 18:31:57.531844 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:31:57.531825    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69\": container with ID starting with 7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69 not found: ID does not exist" containerID="7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69"
Oct 30 18:31:57.531925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.531867    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69"} err="failed to get container status \"7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69\": rpc error: code = NotFound desc = could not find container \"7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69\": container with ID starting with 7b11ce53abdfd9cf2f2d040345c595eee0cd87e3be83c4b37b7c5a05fa0b2d69 not found: ID does not exist"
Oct 30 18:31:57.531925 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.531884    5534 scope.go:122] "RemoveContainer" containerID="7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084"
Oct 30 18:31:57.539680 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.539633    5534 scope.go:122] "RemoveContainer" containerID="7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084"
Oct 30 18:31:57.539955 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:31:57.539938    5534 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084\": container with ID starting with 7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084 not found: ID does not exist" containerID="7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084"
Oct 30 18:31:57.540009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.539965    5534 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084"} err="failed to get container status \"7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084\": rpc error: code = NotFound desc = could not find container \"7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084\": container with ID starting with 7d7c19142b8d7110a4951aeee63bdb02a79797f4abc2c9fad8ac0cac7fade084 not found: ID does not exist"
Oct 30 18:31:57.569105 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.569073    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"]
Oct 30 18:31:57.575083 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.575063    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lmw9v"]
Oct 30 18:31:57.637353 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.637290    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"]
Oct 30 18:31:57.641278 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.641259    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j4pdf"]
Oct 30 18:31:57.674956 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.674929    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"]
Oct 30 18:31:57.678828 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.678807    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8ttz7"]
Oct 30 18:31:57.713277 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.713249    5534 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"]
Oct 30 18:31:57.720726 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:57.720695    5534 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-92jb6"]
Oct 30 18:31:58.367917 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:58.367883    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5757d303-8049-4309-8652-001808326e91" path="/var/lib/kubelet/pods/5757d303-8049-4309-8652-001808326e91/volumes"
Oct 30 18:31:58.368363 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:58.368316    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" path="/var/lib/kubelet/pods/cab59453-7b05-46b1-b731-cdf6fb511b38/volumes"
Oct 30 18:31:58.368710 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:58.368697    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" path="/var/lib/kubelet/pods/e5a9f812-5e0d-4fb8-ab84-77d523901927/volumes"
Oct 30 18:31:58.369453 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:31:58.369438    5534 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" path="/var/lib/kubelet/pods/f3be3002-8b68-42c9-8a9a-74e2774e277b/volumes"
Oct 30 18:32:15.243576 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:32:15.243537    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:32:32.216059 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:32.216009    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:32:45.244338 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:32:45.244299    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:32:49.182892 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:49.182856    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bb675f547-mb27n_4eed760b-e4dc-4f18-aa57-a58ff18df081/manager/0.log"
Oct 30 18:32:49.532483 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:49.532395    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bb675f547-mb27n_4eed760b-e4dc-4f18-aa57-a58ff18df081/manager/0.log"
Oct 30 18:32:50.788399 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:50.788355    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:32:52.129211 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:52.129173    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:32:55.535820 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:55.535736    5534 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log" limit=1024
Oct 30 18:32:55.544334 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:55.544295    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v8rgj_f3dcc262-c93f-43f7-ba5f-601ac5ab06b3/global-pull-secret-syncer/0.log"
Oct 30 18:32:56.152010 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:56.151975    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4hzkr_d255144b-ecb7-489b-98ca-a188acb6c109/konnectivity-agent/0.log"
Oct 30 18:32:56.599033 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:32:56.599001    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-170.ec2.internal_cd29d125f430f9e0a8a7b7b6580a8dec/haproxy/0.log"
Oct 30 18:33:02.970082 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:02.970042    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bc8c78ff8-kx77f_cb838fb8-f499-4b7f-a2ff-99c126affe86/authorino/0.log"
Oct 30 18:33:03.528869 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:03.528841    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-26cq7_7ba1c859-3d6d-47d5-8ab1-72cba22b2146/kuadrant-console-plugin/0.log"
Oct 30 18:33:03.699781 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:03.699747    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-v2kxp_83df9c92-08ea-41d1-9259-c555d8438658/manager/0.log"
Oct 30 18:33:04.679126 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679090    5534 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"]
Oct 30 18:33:04.679698 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679678    5534 cpu_manager.go:414] "containerMap: removing container" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679699    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679715    5534 cpu_manager.go:414] "containerMap: removing container" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679722    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679734    5534 cpu_manager.go:414] "containerMap: removing container" podUID="4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" containerName="maas-api"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679742    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" containerName="maas-api"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679759    5534 cpu_manager.go:414] "containerMap: removing container" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679767    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679784    5534 cpu_manager.go:414] "containerMap: removing container" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy"
Oct 30 18:33:04.679796 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679790    5534 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy"
Oct 30 18:33:04.680487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679890    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab59453-7b05-46b1-b731-cdf6fb511b38" containerName="istio-proxy"
Oct 30 18:33:04.680487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679902    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a9f812-5e0d-4fb8-ab84-77d523901927" containerName="istio-proxy"
Oct 30 18:33:04.680487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679911    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="4688e222-f7ce-4b0d-89ea-b4f88f7fa2b0" containerName="maas-api"
Oct 30 18:33:04.680487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679920    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3be3002-8b68-42c9-8a9a-74e2774e277b" containerName="istio-proxy"
Oct 30 18:33:04.680487 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.679947    5534 memory_manager.go:365] "RemoveStaleState removing state" podUID="5757d303-8049-4309-8652-001808326e91" containerName="istio-proxy"
Oct 30 18:33:04.683144 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.683124    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.698631 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.698609    5534 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cjt4j\"/\"default-dockercfg-pcvs7\""
Oct 30 18:33:04.712201 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.712172    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"]
Oct 30 18:33:04.715052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.714987    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-podres\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.715052 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.715031    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcgk\" (UniqueName: \"kubernetes.io/projected/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-kube-api-access-5rcgk\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.715307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.715173    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-sys\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.715307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.715237    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjt4j\"/\"kube-root-ca.crt\""
Oct 30 18:33:04.715307 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.715250    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-proc\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.716009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.715359    5534 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-lib-modules\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.716009 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.715403    5534 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjt4j\"/\"openshift-service-ca.crt\""
Oct 30 18:33:04.816675 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816598    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-sys\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816713    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-proc\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816736    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-sys\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816738    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-lib-modules\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816809    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-lib-modules\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816835    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-proc\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816861    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-podres\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.816890 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816882    5534 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcgk\" (UniqueName: \"kubernetes.io/projected/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-kube-api-access-5rcgk\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.817172 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.816950    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-podres\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.840359 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.840320    5534 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcgk\" (UniqueName: \"kubernetes.io/projected/3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74-kube-api-access-5rcgk\") pod \"perf-node-gather-daemonset-qqp6n\" (UID: \"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:04.994920 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:04.994835    5534 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:05.294417 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:05.294342    5534 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"]
Oct 30 18:33:05.297494 ip-10-0-135-170 kubenswrapper[5534]: W1030 18:33:05.297464    5534 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3b7a4b12_b7f3_4f1e_a8fa_b8000f58dd74.slice/crio-3cb2938d7a2e92bfd7d30a23e366cf44498e335c1be199e2ee1783e0fd8bca88 WatchSource:0}: Error finding container 3cb2938d7a2e92bfd7d30a23e366cf44498e335c1be199e2ee1783e0fd8bca88: Status 404 returned error can't find the container with id 3cb2938d7a2e92bfd7d30a23e366cf44498e335c1be199e2ee1783e0fd8bca88
Oct 30 18:33:05.814508 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:05.814462    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n" event={"ID":"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74","Type":"ContainerStarted","Data":"cb01d0c8fb8f84d2b1b6a4ac4f49ab17c88c6c189a3bfbfe2e66aa720be03aa0"}
Oct 30 18:33:05.815008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:05.814532    5534 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n" event={"ID":"3b7a4b12-b7f3-4f1e-a8fa-b8000f58dd74","Type":"ContainerStarted","Data":"3cb2938d7a2e92bfd7d30a23e366cf44498e335c1be199e2ee1783e0fd8bca88"}
Oct 30 18:33:05.815008 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:05.814601    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:05.854485 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:05.854437    5534 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n" podStartSLOduration=1.854423398 podStartE2EDuration="1.854423398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:05.853736709 +0000 UTC m=+4791.615780523" watchObservedRunningTime="2026-10-30 18:33:05.854423398 +0000 UTC m=+4791.616467228"
Oct 30 18:33:06.530745 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:06.530721    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qczgg_908859a0-1e77-4a62-a253-3486242475d8/kube-state-metrics/0.log"
Oct 30 18:33:06.652294 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:06.652268    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qczgg_908859a0-1e77-4a62-a253-3486242475d8/kube-rbac-proxy-main/0.log"
Oct 30 18:33:06.779569 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:06.779542    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qczgg_908859a0-1e77-4a62-a253-3486242475d8/kube-rbac-proxy-self/0.log"
Oct 30 18:33:07.047775 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:07.047746    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-mxs5v_df1c926e-5a6b-4c44-881c-5dc47879a022/monitoring-plugin/0.log"
Oct 30 18:33:07.928691 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:07.928656    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbfgv_322b7165-7398-4ed9-99a9-cef5e8ee3c86/node-exporter/0.log"
Oct 30 18:33:08.057978 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.057951    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbfgv_322b7165-7398-4ed9-99a9-cef5e8ee3c86/kube-rbac-proxy/0.log"
Oct 30 18:33:08.183310 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.183248    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbfgv_322b7165-7398-4ed9-99a9-cef5e8ee3c86/init-textfile/0.log"
Oct 30 18:33:08.314250 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.314221    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-kdm8l_376a21c0-10de-43a4-a04c-c3519a868a9b/kube-rbac-proxy-main/0.log"
Oct 30 18:33:08.445758 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.445677    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-kdm8l_376a21c0-10de-43a4-a04c-c3519a868a9b/kube-rbac-proxy-self/0.log"
Oct 30 18:33:08.573002 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.572973    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-kdm8l_376a21c0-10de-43a4-a04c-c3519a868a9b/openshift-state-metrics/0.log"
Oct 30 18:33:08.713952 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.713882    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/prometheus/0.log"
Oct 30 18:33:08.838072 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.838044    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/config-reloader/0.log"
Oct 30 18:33:08.964206 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:08.964132    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/thanos-sidecar/0.log"
Oct 30 18:33:09.096197 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:09.096170    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/kube-rbac-proxy-web/0.log"
Oct 30 18:33:09.221267 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:09.221190    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/kube-rbac-proxy/0.log"
Oct 30 18:33:09.348387 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:09.348357    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/kube-rbac-proxy-thanos/0.log"
Oct 30 18:33:09.476530 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:09.476459    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dca0b75-8ca5-4e65-8c40-75ca747fffb3/init-config-reloader/0.log"
Oct 30 18:33:10.003018 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:10.002988    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85f86dfd46-lsb6t_4c86d1f6-23f6-4d70-bcac-400d7c08bc09/telemeter-client/0.log"
Oct 30 18:33:10.127253 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:10.127222    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85f86dfd46-lsb6t_4c86d1f6-23f6-4d70-bcac-400d7c08bc09/reload/0.log"
Oct 30 18:33:10.252759 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:10.252735    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85f86dfd46-lsb6t_4c86d1f6-23f6-4d70-bcac-400d7c08bc09/kube-rbac-proxy/0.log"
Oct 30 18:33:11.829447 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:11.829420    5534 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-qqp6n"
Oct 30 18:33:15.243422 ip-10-0-135-170 kubenswrapper[5534]: E1030 18:33:15.243386    5534 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p274z" containerName="extractor"
Oct 30 18:33:18.873414 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:18.873387    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcllv_a6a516e5-f016-4d54-be70-b1979fe0c605/dns/0.log"
Oct 30 18:33:18.997222 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:18.997157    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcllv_a6a516e5-f016-4d54-be70-b1979fe0c605/kube-rbac-proxy/0.log"
Oct 30 18:33:19.376086 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:19.376061    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zz6f9_991f66af-6bcb-4bd8-921f-58f2fb879067/dns-node-resolver/0.log"
Oct 30 18:33:19.775413 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:19.775328    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57db9d4f85-fc579_e8a85129-d6fd-42c0-b2a3-50187e0edf0a/kube-auth-proxy/0.log"
Oct 30 18:33:20.855270 ip-10-0-135-170 kubenswrapper[5534]: I1030 18:33:20.855236    5534 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xjh5h_7a3c7100-3ff5-4410-abd7-03e68c27baa3/node-ca/0.log"